yihua opened a new pull request, #12490: URL: https://github.com/apache/hudi/pull/12490
### Change Logs This PR fixes the bootstrap support with file group reader in compaction and clustering in Spark. While the file group reader already supports reading bootstrap table fully, the wiring in compaction and clustering had issues before this PR: (1) The file size is unknown based on the compaction or clustering plan. When calling file group reader, the file size is not properly populated to the reader, causing no rows to be read in the case of bootstrapped file groups. (2) Clustering execution does not prepare the file slice with bootstrap file. This PR contains the following changes to address these issues: - Fixes the file group reader to determine the base file size if the length is negative so that it does not cause the reader to skip reading the file; - Passes `-1` as the base file length when calling the file group reader from `HoodieCompactor` (compaction) and `HoodieSparkFileGroupReaderBasedMergeHandle` (clustering) so that the file group reader can automatically determine the size (the compaction and clustering plans do not have the file length information, unlike the file info provided by the file system view); - Removes the check on bootstrap file groups in `MultipleSparkJobExecutionStrategy` for enable clustering using file group reader on boostrap tables in Spark; - Passes in boostrap file, if present, to the file group reader in `MultipleSparkJobExecutionStrategy`; - Removes the check on bootstrap file groups in `HoodieCompactor` for enable compaction using file group reader on boostrap tables in Spark; - Adds more test coverage in `TestHoodieFileGroupReaderBase`, `TestHoodieFileGroupReaderOnSpark`, improves data validation in `TestDataSourceForBootstrap`; these tests, particularly `TestDataSourceForBootstrap#testMetadataBootstrapMORPartitionedInlineClustering` and `TestDataSourceForBootstrap#testMetadataBootstrapMORPartitionedInlineCompactionOn`, failed without the fix. ### Impact As above for fixing functionality gap with file group reader in compaction and clustering in Spark ### Risk level none ### Documentation Update N/A ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
