acrites commented on code in PR #40639:
URL: https://github.com/apache/beam/pull/40639#discussion_r4327812520
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/work/processing/ComputationWorkExecutorFactory.java:
##########
@@ -166,23 +168,27 @@ private static void trackAutoscalingBytesRead(
MapTask mapTask,
Nodes.ParallelInstructionNode readNode,
Coder<?> readCoder,
- ReadOperation readOperation,
+ Operation operation,
MapTaskExecutor mapTaskExecutor,
String counterName) {
- NameContext nameContext =
- NameContext.create(
- mapTask.getStageName(),
- readNode.getParallelInstruction().getOriginalName(),
- readNode.getParallelInstruction().getSystemName(),
- readNode.getParallelInstruction().getName());
- readOperation.receivers[0].addOutputCounter(
- counterName,
- new OutputObjectAndByteCounter(
- new
IntrinsicMapTaskExecutorFactory.ElementByteSizeObservableCoder<>(readCoder),
- mapTaskExecutor.getOutputCounters(),
- nameContext)
- .setSamplingPeriod(100)
- .countBytes(counterName));
+ if (operation.receivers != null
+ && operation.receivers.length > 0
+ && operation.receivers[0] != null) {
+ NameContext nameContext =
+ NameContext.create(
+ mapTask.getStageName(),
+ readNode.getParallelInstruction().getOriginalName(),
+ readNode.getParallelInstruction().getSystemName(),
+ readNode.getParallelInstruction().getName());
+ operation.receivers[0].addOutputCounter(
Review Comment:
I'm not entirely sure. I didn't actually change this code, just added some
extra null checks. I guess as long as the SplittableFn is not outputting the
same element across multiple outputs we don't risk double-counting. I think
that might more closely match what the portable worker does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
