ahmedabu98 commented on code in PR #38018:
URL: https://github.com/apache/beam/pull/38018#discussion_r3018253494
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFiles.java:
##########
@@ -205,10 +230,15 @@ static class ConvertToDataFile extends DoFn<String,
SerializableDataFile> {
private final @Nullable String prefix;
private final @Nullable List<String> partitionFields;
private final @Nullable Map<String, String> tableProps;
+ private transient @MonotonicNonNull ExecutorService executor;
+ private transient @MonotonicNonNull List<Future<ProcessResult>>
activeTasks;
private transient @MonotonicNonNull Table table;
+
// Limit open readers to avoid blowing up memory on one worker
private static final int MAX_READERS = 10;
private static final Semaphore ACTIVE_READERS = new Semaphore(MAX_READERS);
+ // Number of parallel threads processing incoming files
+ private static final int THREAD_POOL_SIZE = 10;
Review Comment:
Doing some experiments now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
