acrites opened a new pull request, #41487:
URL: https://github.com/apache/beam/pull/41487

   Track custom source bytes read directly on StreamingModeExecutionContext via 
a dedicated ElementByteSizeCounter instead of registering a shared Counter in 
pendingDeltaCounters.
   
   Previously, ComputationWorkExecutorFactory registered 
'dataflow_source_bytes_processed-<systemName>' in the worker-wide 
pendingDeltaCounters CounterSet. This caused two races on getAndReset():
   1. StreamingWorkerStatusReporter periodically extracts and resets all 
modified delta counters in pendingDeltaCounters, draining source bytes before 
WorkItemCommitRequest is built.
   2. Concurrent ComputationWorkExecutors for the same computation shared a 
single Counter instance by name and raced on getAndReset().
   
   Changes
   1. Add ElementByteSizeCounter, an ElementCounter that observes encoded 
element byte sizes (with adaptive sampling and lazy coder support) and reports 
them to a LongConsumer without registering in a CounterSet. Otherwise, sampling 
implementation is the same as OutputObjectAndByteCounter.
   2. Track sourceBytesProcessed per execution context on 
StreamingModeExecutionContext, resetting per key/commit and on reset().
   3. Remove the unused sourceBytesProcessCounterName plumbing from 
ComputationState and StreamingModeExecutionContext.
   4. Add unit tests in ElementByteSizeCounterTest and 
StreamingModeExecutionContextTest, plus an end-to-end regression test in 
StreamingDataflowWorkerTest covering mid-bundle delta counter extraction and 
concurrent source work items.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to