Anurag-S1ngh opened a new pull request, #41358:
URL: https://github.com/apache/beam/pull/41358
## Summary
`BundleBasedDirectRunner` creates a fresh `MetricsContainer` for every
bundle but reuses the state sampler's cached `ScopedState` for a step. The
cached state kept the first bundle's `MetricsContainer`, so counter increments
made in later bundles were never recorded in `PipelineResult.metrics()`, and
metrics first used in a later bundle were missing entirely.
The state sampler now refreshes the cached state's container on a cache hit.
`ScopedState` gets a `set_metrics_container` method in both the slow and the
Cython sampler; the Cython attribute stays readonly so callers keep going
through that method.
Fixes #41352
## Testing
- New `BundleBasedRunnerTest.test_metrics_across_multiple_bundles`: a
`TestStream` sends elements in separate bundles and asserts both counters,
which fails before the change (`{'seen': 1}`) and passes after (`{'seen': 2,
'second_bundle': 1}`).
- `pytest sdks/python/apache_beam/runners/worker/statesampler_test.py
sdks/python/apache_beam/runners/direct/direct_runner_test.py::BundleBasedRunnerTest`
(8 passed) with both the slow sampler and a freshly compiled
`statesampler_fast` extension.
- The issue reproducer returns `{'seen': 2, 'second_bundle': 1}` with both
sampler implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]