junaiddshaukat opened a new pull request, #39341:
URL: https://github.com/apache/beam/pull/39341
Collects the MonitoringInfos the SDK harness reports with each bundle and
exposes them as Beam MetricResults. Part of #18479, first step toward the
@ValidatesRunner suite -- PAssert verifies its assertions ran via a success
counter it reads from PipelineResult.metrics(), which previously threw
not-implemented.
Follows the Flink pattern (FlinkExecutableStageFunction):
- The translation context owns a MetricsContainerStepMap, one container per
executable stage.
- ExecutableStageProcessor replaces BundleProgressHandler.ignored() with a
handler that folds the harness's MonitoringInfos into its stage's
container on
bundle progress and on completion.
- KafkaStreamsPortablePipelineResult.metrics() returns
asAttemptedOnlyMetricResults; attempted values only for now -- committed
metrics need to be folded into the exactly-once commit, which belongs with
the
durability work.
- KafkaStreamsTestRunner.run now returns the MetricResults so tests can
assert
on user counters.
Test: MetricsTest runs Create.of(1, 2, 3) through a counting DoFn in the
EMBEDDED harness and asserts the user counter reads 3 through the runner's
MetricResults.
Next up (separate PRs): a TestPipeline-dispatchable runner harness, then the
validatesRunner gradle task with the first enabled tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
