claudevdm commented on code in PR #39896:
URL: https://github.com/apache/beam/pull/39896#discussion_r3914936693
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFiles.java:
##########
@@ -329,50 +314,33 @@ private static class ProcessResult {
@Setup
public void setup() {
- executor = Executors.newFixedThreadPool(THREAD_POOL_SIZE);
+ tasks = new BoundedAsyncTasks<>(THREAD_POOL_SIZE, MAX_IN_FLIGHT_TASKS);
+ }
+
+ /** Clears anything left behind if the runner reuses this instance after a
failed bundle. */
+ @StartBundle
+ public void startBundle() {
+
+ checkStateNotNull(tasks).cancelAll();
}
@Teardown
public void teardown() {
- if (executor != null) {
- executor.shutdownNow();
+ if (tasks != null) {
+ tasks.shutdown();
}
}
- @StartBundle
- public void startBundle() {
- activeTasks = Lists.newLinkedList();
- }
-
@ProcessElement
public void process(
@Element String filePath,
@Timestamp Instant timestamp,
BoundedWindow window,
PaneInfo paneInfo,
MultiOutputReceiver output)
- throws IOException, InterruptedException, ExecutionException {
- LinkedList<Future<ProcessResult>> activeTasks =
checkStateNotNull(this.activeTasks);
-
- // start draining finished tasks, but don't block
- Iterator<Future<ProcessResult>> iterator = activeTasks.iterator();
- while (iterator.hasNext()) {
- Future<ProcessResult> future = iterator.next();
- if (future.isDone()) {
- outputResult(future.get(), output);
- iterator.remove();
- }
- }
-
- // if we have too many active tasks, wait until some finish
- while (activeTasks.size() >= MAX_IN_FLIGHT_TASKS) {
- Future<ProcessResult> oldestTask = activeTasks.removeFirst();
- outputResult(oldestTask.get(), output); // .get() blocks until the
task completes
- }
-
- // create a new task for the current element and add to queue
+ throws Exception {
Callable<ProcessResult> task = createProcessTask(filePath, timestamp,
window, paneInfo);
- activeTasks.add(checkStateNotNull(executor).submit(task));
+ checkStateNotNull(tasks).submit(task, result -> outputResult(result,
output));
Review Comment:
The consumer is invoked synchronously inside submit() and awaitAll() on the
calling thread
https://github.com/claudevdm/beam/blob/f7edad1f06bc04dcd11b1eaee6dad3618b861048/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/BoundedAsyncTasks.java#L62
https://github.com/claudevdm/beam/blob/f7edad1f06bc04dcd11b1eaee6dad3618b861048/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/BoundedAsyncTasks.java#L104-L110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
