acrites opened a new pull request, #41487: URL: https://github.com/apache/beam/pull/41487
Track custom source bytes read directly on StreamingModeExecutionContext via a dedicated ElementByteSizeCounter instead of registering a shared Counter in pendingDeltaCounters. Previously, ComputationWorkExecutorFactory registered 'dataflow_source_bytes_processed-<systemName>' in the worker-wide pendingDeltaCounters CounterSet. This caused two races on getAndReset(): 1. StreamingWorkerStatusReporter periodically extracts and resets all modified delta counters in pendingDeltaCounters, draining source bytes before WorkItemCommitRequest is built. 2. Concurrent ComputationWorkExecutors for the same computation shared a single Counter instance by name and raced on getAndReset(). Changes 1. Add ElementByteSizeCounter, an ElementCounter that observes encoded element byte sizes (with adaptive sampling and lazy coder support) and reports them to a LongConsumer without registering in a CounterSet. Otherwise, sampling implementation is the same as OutputObjectAndByteCounter. 2. Track sourceBytesProcessed per execution context on StreamingModeExecutionContext, resetting per key/commit and on reset(). 3. Remove the unused sourceBytesProcessCounterName plumbing from ComputationState and StreamingModeExecutionContext. 4. Add unit tests in ElementByteSizeCounterTest and StreamingModeExecutionContextTest, plus an end-to-end regression test in StreamingDataflowWorkerTest covering mid-bundle delta counter extraction and concurrent source work items. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
