prashantwason opened a new pull request, #20555:
URL: https://github.com/apache/hudi/pull/20555
### Describe the issue this Pull Request addresses
With
`hoodie.datasource.read.file.index.list.file.statuses.using.ro.path.filter=true`,
a Spark snapshot read of a METADATA_ONLY bootstrapped COW table returns the
correct row count, but every data column is null. Only the skeleton meta
columns are populated.
This was found on a real production table: the row count matched the source,
but a content hash showed every data column read as null.
The option was added by #18136 ("perf: Adding support for
LatestBaseFilesPathFilter to Spark File Index"). With it enabled,
`BaseHoodieTableFileIndex.loadFileSlicesForPartitions` takes the
`generatePartitionFileSlicesPostROTablePathFilter` fast path for COW snapshot
and read-optimized queries. That path builds the file slices directly from the
listed files, and the base files it creates lack their bootstrap base file. The
skeleton file is read without the bootstrap source file that holds the data
columns.
### Summary and Changelog
- `BaseHoodieTableFileIndex`: do not take the RO path filter fast path when
the table has a bootstrap base path
(`metaClient.getTableConfig().getBootstrapBasePath().isPresent()`).
Bootstrapped tables fall back to the regular `filterFiles` path, which keeps
the bootstrap base files.
- `TestBootstrapRead#testMetadataBootstrapReadWithPathFilterListing`: new
parameterized test (unpartitioned and 2-level partitioned) that bootstraps a
COW table with the metadata bootstrap type, enables the RO path filter listing
through the `spark.`-prefixed session conf, and compares the result against the
source table.
Verified locally (Spark 3.5 profile):
- The new test fails without the main-code change (`expected: <0> but was:
<75>`, both parameters) and passes with it.
- All of `TestBootstrapRead` passes (5 tests, 0 failures).
- `checkstyle:check` passes on the built modules.
### Impact
Fixes incorrect results (null data columns) for bootstrapped tables when the
RO path filter listing option is enabled. The option is off by default, so
default behavior is unchanged. Bootstrapped tables with the option enabled lose
the listing optimization and use the regular listing path.
### Risk Level
low. The change only adds a condition that skips an optional fast path for
tables with a bootstrap base path. Non-bootstrapped tables are unaffected.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
