junaiddshaukat commented on code in PR #39341:
URL: https://github.com/apache/beam/pull/39341#discussion_r3588490592
##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/translation/KafkaStreamsTranslationContext.java:
##########
@@ -45,6 +46,10 @@ public class KafkaStreamsTranslationContext {
private final KafkaStreamsPipelineOptions pipelineOptions;
private final Topology topology;
private final Map<String, String> pCollectionIdToProcessorName;
+ // Accumulates the Beam metrics reported by the SDK harness, one container
per executable stage.
+ // Processors update it as bundles complete (in-JVM reference sharing); the
pipeline result
+ // exposes it as MetricResults.
+ private final MetricsContainerStepMap metricsContainerStepMap = new
MetricsContainerStepMap();
Review Comment:
The premise here isn't right: MetricsContainerImpl.update() applies counters
via CounterCell.inc() (add, not set), and the cells are thread-safe (AtomicLong
counters, CAS distributions, backed by ConcurrentHashMap). Since each update is
a bundle's final values, concurrent tasks of a stage accumulate correctly in
one shared container — same reason a per-task split isn't needed. The real
limitation in this area is aggregation across multiple runner JVMs once
instances scale out, which a per-task list inside one JVM wouldn't address
either; that belongs to the multi-instance work. Added a comment documenting
this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
