lostluck commented on issue #19632: URL: https://github.com/apache/beam/issues/19632#issuecomment-1650381180
Dataflow side just discussed this a bit, so forwarding some key points: Beam Metrics over the FnAPI are always qualified by Transform, Namespace, Name, and additionally by Bundle. That last one is ephemeral, and used to get precise counters WRT bundle retries. So in that respect, a runner can already always "know" which transform a counter comes from, and then act accordingly. But for the most part, there's no reason to distinguish between transforms that are throttling. The key part then is deciding officially on a common "beam" namespace (I'd suggest `beam` or `beam:standard:counter`, over something javalike `org.apache.beam.standard.count`) (I'll go with `beam` for now) and a name for the counter itself, which might as well be `throttling-msecs` since there are several IOs already doing that. So the hard part of this (aside from a dev list community discussion of the proposal) would be clearly documenting that the `beam` namespace has certain reserved counter names, that runners may take special note of for whatever they need. Throttling is one case here. But even things like `rpc-count`, `rpc-retries`, etc etc... And this should be documented clearly in the "IO" guide, and any SDK building guide on the beam website. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
