Anurag-S1ngh opened a new pull request, #41358:
URL: https://github.com/apache/beam/pull/41358

   ## Summary
   
   `BundleBasedDirectRunner` creates a fresh `MetricsContainer` for every 
bundle but reuses the state sampler's cached `ScopedState` for a step. The 
cached state kept the first bundle's `MetricsContainer`, so counter increments 
made in later bundles were never recorded in `PipelineResult.metrics()`, and 
metrics first used in a later bundle were missing entirely.
   
   The state sampler now refreshes the cached state's container on a cache hit. 
`ScopedState` gets a `set_metrics_container` method in both the slow and the 
Cython sampler; the Cython attribute stays readonly so callers keep going 
through that method.
   
   Fixes #41352
   
   ## Testing
   
   - New `BundleBasedRunnerTest.test_metrics_across_multiple_bundles`: a 
`TestStream` sends elements in separate bundles and asserts both counters, 
which fails before the change (`{'seen': 1}`) and passes after (`{'seen': 2, 
'second_bundle': 1}`).
   - `pytest sdks/python/apache_beam/runners/worker/statesampler_test.py 
sdks/python/apache_beam/runners/direct/direct_runner_test.py::BundleBasedRunnerTest`
 (8 passed) with both the slow sampler and a freshly compiled 
`statesampler_fast` extension.
   - The issue reproducer returns `{'seen': 2, 'second_bundle': 1}` with both 
sampler implementations.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] [Python] Update cached Sc... Anurag-S1ngh (via GitHub) via github

Reply via email to