vamsikarnika opened a new pull request, #951: URL: https://github.com/apache/incubator-xtable/pull/951
## What Found while syncing a real 24-table, 1TB, thousands-of-partitions-per-table Hudi dataset to Iceberg via `RunSync`. `tryEnrichWithMetadataStats()` accepted the Hudi metadata table's column-stats entry for a file as valid whenever the entry existed at all — even when every column's value count came back non-positive. In that case it silently emits `recordCount=0` for a file that has real data, instead of falling back to the (already-existing) Parquet-footer-based stats path, which is only triggered when the metadata table has **no** entry at all, not when it has an entry with unusable values. Observed on real files written before column-stats indexing was enabled for their table: correct data, correct row counts in the actual Parquet footers, but zero-valued entries in the metadata table's index — resulting in `recordCount=0` (and empty column stats) written into the synced Iceberg table's manifest. ## Fix Treat an absent/non-positive max the same as "no stats" and fall back to the footer-based path. Added a regression test reproducing the exact scenario (metadata table returns a non-empty entry with `valueCount=0` for every column; asserts the result falls back to the real row count from the Parquet footer). ## Testing - New unit test `columnStatsWithMetadataTableZeroValueCountFallsBackToParquetFooters` passes. - Full `xtable-core` module compiles and `TestHudiFileStatsExtractor` suite passes against the real, publicly-published `org.apache.hudi:hudi-common:1.2.0` (verified with a from-scratch `~/.m2` resolution — one pre-existing, unrelated failure in `columnStatsWithMetadataTable`, a `NoSuchFieldError` deep in Hudi's own bulk-insert internals on a code path this PR doesn't touch). - Manually verified end-to-end on real data: synced all 24 tables of a real 1TB TPC-DS Hudi dataset to Iceberg; before this fix, non-partitioned dimension tables showed `COUNT(*) = 0` via both Trino's and Presto's Iceberg connectors despite having real data; after the fix, all 24 tables report correct row counts matching the source. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DFemyCQKjw9RTX3P3vixY1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
