scwhittle commented on code in PR #25338:
URL: https://github.com/apache/beam/pull/25338#discussion_r1097853288
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java:
##########
@@ -219,6 +219,10 @@ public static void main(
ShortIdMap metricsShortIds = new ShortIdMap();
ExecutorService executorService =
options.as(ExecutorOptions.class).getScheduledExecutorService();
+ // In order to reduce memory spent on per-thread coders and buffers, we
separate finalizations
+ // from standard bundle processing.
+ ExecutorService finalizationExecutorService =
Review Comment:
The ones I noticed were related to AvroCoder.
There is a thread-local for the binarycoder here:
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java#L299
and thread-local for GenericReader here:
https://github.com/rdblue/avro-java/blob/master/avro/src/main/java/org/apache/avro/generic/GenericDatumReader.java#L121
This was in Spanner->Avro GCS template pipeline. So we might be able to
change the coder there to see gains. But it still seems like a generic issue
with a single thread pool with distinct work types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
