claudevdm commented on code in PR #39896:
URL: https://github.com/apache/beam/pull/39896#discussion_r3873862963
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFiles.java:
##########
@@ -329,50 +314,33 @@ private static class ProcessResult {
@Setup
public void setup() {
- executor = Executors.newFixedThreadPool(THREAD_POOL_SIZE);
+ tasks = new BoundedAsyncTasks<>(THREAD_POOL_SIZE, MAX_IN_FLIGHT_TASKS);
Review Comment:
The logic for back-pressure (process block when 100 tasks outstanding,
handing back results on main thread, bookkeeping) was inline in the
ConvertToDatafile DoFn. I extracted it to make it testable in isolation and
reusable by other DoFn than ConvertToDatafile (it will also be needed in the
schema prepass when reading footers).
Lets consider moving it to sdk/util in a follow up, I think that would
require a refactor and some more thought.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
