yihua opened a new pull request, #20263: URL: https://github.com/apache/hudi/pull/20263
### Describe the issue this Pull Request addresses closes #20262 part of #20257 The bloom index key check resolves each candidate base file through the file system view twice per handle, which on executors means two timeline-server `getLatestBaseFile` requests per (task, candidate base file), although the driver already resolved those base files when it loaded the key ranges. The bloom, key-check and SIMPLE index reads also look up each parquet file's status before opening it, although its size is known. The two lookups per handle date from #726 (0.5.0), which resolved the base file through the table view in both the `HoodieKeyLookupHandle` constructor and `getLookupResult`; they became timeline-server requests when the remote view became the default (#1634, 0.5.3). #6793 (HUDI-4917, 0.13.0) already applied "the driver has the base file" to the range-info stage but not to the key check. ### Summary and Changelog The key-check stage now reads the base files the driver resolved instead of asking the view for them, and parquet footer and key reads skip the status lookup when the file length is known. - `BloomIndexFileInfo` carries the resolved `HoodieBaseFile` (not part of equality); the loaders set it, and both bloom index helpers pass a file-group-to-base-file lookup (a broadcast on Spark) to `HoodieBloomIndexCheckFunction`. `ListBasedIndexFileFilter` keeps only the key ranges, so the base files are not shipped with the record-to-file pairing tasks. - `HoodieKeyLookupHandle` takes an optional base file and falls back to a single view lookup when it is absent. - `FileFormatUtils` gains `StoragePathInfo` overloads for footer, bloom filter, min/max key, key filter and key iterator reads; `ParquetUtils` opens the file from its known length (with the read options of `withHadoopReadOptions`), and the Avro and Spark parquet readers and factories use them. `BaseFile#getPathInfoWithKnownSize` feeds them from the read handle, `HoodieIndexUtils.filterKeysFromFile` and `HoodieKeyLocationFetchHandle`. `HoodieFileReaderFactory.getFileReader(config, pathInfo, PARQUET, ...)` now builds the known-length reader for its existing callers too (e.g. `HoodieAvroReaderContext`); only the footer, bloom filter and key reads use the length. - Tests: `TestHoodieBloomIndex#testKeyCheckReadsResolvedBaseFiles` (BLOOM and GLOBAL_BLOOM, 9 helper and metadata configurations) asserts no timeline-server base-file request and no parquet status call during tagging, and that locations match the SIMPLE index; it fails on master. `TestHoodieKeyLocationFetchHandle` asserts the status calls, and `TestParquetUtils` covers the new overloads. ### Impact Fewer executor round trips for BLOOM, GLOBAL_BLOOM and SIMPLE index lookups, with no change to results. On a local Spark harness, executor timeline-server requests from the key check go from 64 to 0 (256 to 0 at 4x file groups per partition), and executor base-file status calls go from 152 to 80 (352 to 80 at 4x file groups); the key check makes none during tagging, and the remaining 80 come from other paths (column stats footer reads, the merge handle and write-stat sizes). On S3A, `fs.open()` still issues a HEAD because parquet's `HadoopInputFile.newStream` does not pass the status, so S3A HEADs per candidate file drop from 2 to 1. `HoodieKeyLookupHandle`, `HoodieBloomIndexCheckFunction` and `HoodieSparkBloomIndexCheckFunction` constructors change (internal classes). This overlaps #20095 in `HoodieKeyLocationFetchHandle.fetchRecordKeysWithPositions`, a trivial conflict whichever lands second. ### Risk Level low: the base file comes from the same driver-side view the range stage already uses, and a wrong recorded file size fails the footer read loudly rather than returning wrong data. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
