junaiddshaukat commented on code in PR #39341:
URL: https://github.com/apache/beam/pull/39341#discussion_r3588505884
##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/KafkaStreamsPipelineRunner.java:
##########
@@ -62,7 +62,8 @@ public PortablePipelineResult run(RunnerApi.Pipeline
pipeline, JobInfo jobInfo)
KafkaStreams kafkaStreams = new KafkaStreams(topology,
streamsConfig(jobInfo));
kafkaStreams.start();
- return new KafkaStreamsPortablePipelineResult(kafkaStreams);
+ return new KafkaStreamsPortablePipelineResult(
+ kafkaStreams, context.getMetricsContainerStepMap());
Review Comment:
Same as the first thread — the shared container is safe; see reply there.
##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/translation/ExecutableStageTranslator.java:
##########
@@ -86,12 +86,17 @@ public void translate(
Topology topology = context.getTopology();
// The stage stamps its own transform id on the watermarks it emits, and
aggregates its input
// watermark from the reports of its single upstream transform (the
producer of its input
- // PCollection, whose node name is the upstream transform id).
+ // PCollection, whose node name is the upstream transform id).
Harness-reported metrics land in
+ // this stage's container of the job's metrics step map.
topology.addProcessor(
transformId,
() ->
new ExecutableStageProcessor(
- stagePayload, context.getJobInfo(), transformId,
ImmutableSet.of(parentProcessor)),
+ stagePayload,
+ context.getJobInfo(),
+ transformId,
+ ImmutableSet.of(parentProcessor),
+
context.getMetricsContainerStepMap().getContainer(transformId)),
Review Comment:
Same as the first thread — the shared container is safe; see reply there.
##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/translation/KafkaStreamsTranslationContext.java:
##########
@@ -77,6 +82,16 @@ public Topology getTopology() {
return topology;
}
+ /**
+ * Returns the job's metrics accumulator: one {@link
+ * org.apache.beam.runners.core.metrics.MetricsContainerImpl container} per
executable stage,
+ * updated by the stage processors as the SDK harness reports bundle
metrics, and read by the
+ * pipeline result via {@link
MetricsContainerStepMap#asAttemptedOnlyMetricResults}.
+ */
+ public MetricsContainerStepMap getMetricsContainerStepMap() {
+ return metricsContainerStepMap;
+ }
Review Comment:
Same as the first thread — the shared container is safe; see reply there.
##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/translation/KafkaStreamsTranslationContext.java:
##########
@@ -45,6 +46,10 @@ public class KafkaStreamsTranslationContext {
private final KafkaStreamsPipelineOptions pipelineOptions;
private final Topology topology;
private final Map<String, String> pCollectionIdToProcessorName;
+ // Accumulates the Beam metrics reported by the SDK harness, one container
per executable stage.
+ // Processors update it as bundles complete (in-JVM reference sharing); the
pipeline result
+ // exposes it as MetricResults.
+ private final MetricsContainerStepMap metricsContainerStepMap = new
MetricsContainerStepMap();
Review Comment:
Same as the first thread — the shared container is safe; see reply there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
