> @ajamato A more general question for my knowledge: as the communication > between runner harness and SDK harness is done on a bundle basis. When the > runner harness sends data to the sdk harness to execute a transform that > contains metrics, does it: > > 1. send metrics values (for the ones defined in the transform) alongside with > data and receive an updated value of the metrics when the bundle is finished > processing? > 2. or does it send only the data and the sdk harness responds with a diff > value of the metrics so that the runner can update them in its side? > > My bet is option 2. But can you confirm?
Option 2, the metric is calculated for the bundle. That is, we send a single metric update for the bundle. The design expects that the upstream systems will handle that aggregation. It may sum all the metrics up together, outside of the SDK. The runner should forward these metrics to a metric aggregation system across to aggregate the metrics to a final value across all the workers. [ Full content available at: https://github.com/apache/beam/pull/6205 ] This message was relayed via gitbox.apache.org for [email protected]
