scwhittle commented on issue #29099: URL: https://github.com/apache/beam/issues/29099#issuecomment-3415574275
Sorry you're right, this still doesn't work e2e. The unit test added in the above PR avoids the problem by setting up a MetricsContainer override but in non-test cases, the metrics will function only if an ExecutionState has been activated for the transform. When we are consuming input data from the data stream this is taken care of by the PCollectionConsumerRegistry before handing off to the consumer. Such ExecutionState activation isn't performed before handing off timers to the timer consumer. I prototyped a solution to expose the existing process bundle execution states from PCollectionConsumerRegistry to ProcessBundleHandler and activating them before forwarding timer data onto FnHarnessRunner.processTimer . This fixed the counters set during timer firing for a reproduction I put together. I will work on cleaning up the change to get it submitted. Potential workarounds in the meantime would be to log or to output (using multiple output PCollections) to a subsequent DoFn which could itself increment the counter. Fusion should keep that processing inline anyway with minimal overhead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
