vinishjail97 opened a new pull request, #961: URL: https://github.com/apache/incubator-xtable/pull/961
## What Addresses the column-stats migration question from the #835 review (https://github.com/apache/incubator-xtable/pull/835#discussion_r4116901814). XTable 0.4.0 did not build the column-stats index for partitioned Hudi targets, and #835 enables it for all tables. On the first sync after the upgrade, Hudi marks `column_stats` available but indexes only the files in that commit. Its bootstrap reads the existing files from the metadata table's `files` partition and drops every name that fails `FSUtils.isDataFile`, which includes all `_hudiext` external files (`HoodieTableMetadataUtil` `DirectoryInfo`). Hudi's Spark reader keeps files without stats, so results stay correct, but the old files get no data skipping. ## Changes - **Detection**: `ConversionTarget#isIncrementalSyncSafe()` lets a target require a snapshot sync. The Hudi target returns false when the table has commits and `isMetadataPartitionAvailable(COLUMN_STATS)` is false, which reads only `hoodie.properties`. - **Re-register**: in that snapshot sync, `BaseFileUpdatesExtractor` writes a new file slice for every snapshot file, including the files the target already has. The slice keeps the fileId and the physical path, so Hudi indexes all files from the commit's write stats. - **Replace guard**: a fileId that the commit writes is never in the replaced file ids, because Hudi hides every slice of a replaced file group. - **Cleaner**: `markInstantsAsCleaned` looked up files to clean only in replaced file groups, so the older slice of a re-registered file threw a `NullPointerException`. It now also looks in live file groups, without the latest slice of each group. ## Tests `ITHudiConversionTarget#snapshotSyncBuildsColStatsForExistingTableWithoutColStats` on table versions 6 and 9: a table with a real parquet file and no column-stats index requires a snapshot sync; after it, both files have stats and the existing file keeps its path; a later clean removes only the older slice and keeps the stats. With the old cleaner lookup the test fails with the `NullPointerException`. Validation: xtable-core 510 unit tests (3 skipped), xtable-api 37; `ITHudiConversionTarget` 20, `ITHudiConversionSource` 51, `ITConversionController` 67. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
