vamsikarnika opened a new pull request, #950: URL: https://github.com/apache/incubator-xtable/pull/950
## What Found while syncing a real 24-table, 1TB, thousands-of-partitions-per-table Hudi dataset to Iceberg via `RunSync`. `getInternalDataFilesForPartitions()` fans out per-partition file listing via a parallel stream, with each partition independently triggering its own metadata-table lookup (and S3 connection) the first time it's touched. For tables with thousands of partitions this exhausts the S3 connection pool and causes lock contention on the shared `FileSystemView`, surfacing as HFile reader crashes, `FileSystemView` lock deadlocks, and connection-pool-exhaustion errors mid-sync. ## Fix Batch-load every partition's metadata in one lookup (`fsView.loadPartitions(partitionPaths)`) before the parallel stream begins — the same pattern `getAddedAndRemovedPartitionInfo()` already uses for incremental commits, just missing from the full-snapshot path. ## Testing Manually verified end-to-end on real data: syncing all 24 tables of a real 1TB TPC-DS Hudi dataset (thousands of partitions on several tables) to Iceberg previously hit connection-pool exhaustion / lock contention partway through; with this fix, the sync completes cleanly for every table with no connection errors, deadlocks, or reader crashes. Full `xtable-core` module compiles and the existing test suite passes against the real, publicly-published `org.apache.hudi:hudi-common:1.2.0` (verified with a from-scratch `~/.m2` resolution). --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DFemyCQKjw9RTX3P3vixY1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
