This is an automated email from the ASF dual-hosted git repository.
vinishjail97 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-xtable.git
The following commit(s) were added to refs/heads/main by this push:
new 8bd19fc0 fix(hudi): Batch-load partitions before the parallel
per-partition file-listing stream (#950)
8bd19fc0 is described below
commit 8bd19fc0c72bd9627032fa2d0e6c57521fb068e1
Author: vamsikarnika <[email protected]>
AuthorDate: Sat Sep 26 04:10:40 2026 +0530
fix(hudi): Batch-load partitions before the parallel per-partition
file-listing stream (#950)
getInternalDataFilesForPartitions() fanned out per-partition file listing
via a parallel stream, with each partition independently triggering its
own metadata-table lookup (and S3 connection) the first time it was
touched. For tables with thousands of partitions this caused S3
connection-pool exhaustion and lock contention on the shared
FileSystemView, surfacing as HFile reader crashes, FileSystemView lock
deadlocks, and connection-pool-exhaustion errors during Hudi-to-Iceberg
sync.
Fix: call fsView.loadPartitions(partitionPaths) once, up front, to
batch-load every partition's metadata in a single lookup before the
parallel stream begins -- the same pattern already used in
getAddedAndRemovedPartitionInfo() for incremental commits, just missing
from the full-snapshot path.
Confirmed: syncing a 24-table, thousands-of-partitions-per-table dataset
to Iceberg now completes cleanly with no connection-pool exhaustion,
lock deadlocks, or reader crashes.
---
.../src/main/java/org/apache/xtable/hudi/HudiDataFileExtractor.java | 6 ++++++
1 file changed, 6 insertions(+)
diff --git
a/xtable-core/src/main/java/org/apache/xtable/hudi/HudiDataFileExtractor.java
b/xtable-core/src/main/java/org/apache/xtable/hudi/HudiDataFileExtractor.java
index a9f2bacc..2a45ae79 100644
---
a/xtable-core/src/main/java/org/apache/xtable/hudi/HudiDataFileExtractor.java
+++
b/xtable-core/src/main/java/org/apache/xtable/hudi/HudiDataFileExtractor.java
@@ -347,6 +347,12 @@ public class HudiDataFileExtractor implements
AutoCloseable {
List<String> partitionPaths, InternalTable table) {
SyncableFileSystemView fsView =
fileSystemViewManager.getFileSystemView(metaClient);
+ // Batch-load all partitions in a single metadata table lookup before
fanning out per
+ // partition below. Without this, the parallel stream below triggers one
metadata table
+ // scan (and one S3 connection) per partition, which for tables with
thousands of
+ // partitions causes S3 connection pool exhaustion and lock contention on
the shared
+ // file system view.
+ fsView.loadPartitions(partitionPaths);
Stream<InternalDataFile> filesWithoutStats =
partitionPaths.stream()
.parallel()
