vamsikarnika opened a new pull request, #949: URL: https://github.com/apache/incubator-xtable/pull/949
**Stacked on #948** (a pre-existing, unrelated compile break on `main` against `hudi.version=1.2.0`) — that PR's diff will disappear from this one once it merges. Found while syncing a real 24-table, 1TB, thousands-of-partitions-per-table Hudi dataset to Iceberg via `RunSync`. Two independent bugs, two commits: ## 1. S3 connection-pool exhaustion / FileSystemView lock deadlocks on high-partition-count tables `getInternalDataFilesForPartitions()` fans out per-partition file listing via a parallel stream, with each partition independently triggering its own metadata-table lookup (and S3 connection) the first time it's touched. For tables with thousands of partitions this exhausts the S3 connection pool and causes lock contention on the shared `FileSystemView`, surfacing as HFile reader crashes, `FileSystemView` lock deadlocks, and connection-pool-exhaustion errors mid-sync. Fix: batch-load every partition's metadata in one lookup (`fsView.loadPartitions(partitionPaths)`) before the parallel stream begins — the same pattern `getAddedAndRemovedPartitionInfo()` already uses for incremental commits, just missing from the full-snapshot path. ## 2. Silent `recordCount=0` when the metadata table has degenerate stats `tryEnrichWithMetadataStats()` accepted the Hudi metadata table's column-stats entry for a file as valid whenever the entry existed at all — even when every column's value count came back non-positive. In that case it silently emits `recordCount=0` for a file that has real data, instead of falling back to the (already-existing) Parquet-footer-based stats path, which is only triggered when the metadata table has **no** entry at all, not when it has an entry with unusable values. Observed on real files written before column-stats indexing was enabled for their table: correct data, correct row counts in the actual Parquet footers, but zero-valued entries in the metadata table's index — resulting in `recordCount=0` (and empty column stats) written into the synced Iceberg table's manifest. Fix: treat an absent/non-positive max the same as "no stats" and fall back to the footer-based path. Added a regression test reproducing the exact scenario (metadata table returns a non-empty entry with `valueCount=0` for every column; asserts the result falls back to the real row count from the Parquet footer). ## Testing - New unit test `columnStatsWithMetadataTableZeroValueCountFallsBackToParquetFooters` passes. - Full existing `TestHudiFileStatsExtractor` suite passes (one pre-existing, unrelated failure in `columnStatsWithMetadataTable` traced to an environment/dependency-version issue in the bulk-insert path, confirmed present before this change too — not something this PR touches). - Manually verified end-to-end: synced all 24 tables of a real 1TB TPC-DS Hudi dataset to Iceberg; before the stats fix, non-partitioned dimension tables showed `COUNT(*) = 0` via Trino/Presto's Iceberg connectors despite having real data; after the fix, all 24 tables report correct row counts matching the source. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DFemyCQKjw9RTX3P3vixY1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
