howardyoo commented on PR #30873: URL: https://github.com/apache/airflow/pull/30873#issuecomment-1524062067
So, I guess this feature is kind of a simple random ’sampling’ mechanism to perhaps control the flow of telemetry data. Interesting. However, I don’t believe we would need this, IMHO. It might even be dangerous to apply rate limiting in such a way. Howard > On Apr 26, 2023, at 4:11 PM, Niko Oliveira ***@***.***> wrote: > > > @o-nikolas commented on this pull request. > > In airflow/metrics/otel_logger.py <https://github.com/apache/airflow/pull/30873#discussion_r1178403467>: > > > + # TODO: I don't think this is the right use for rate??? > + value = count * rate > I'm not sure how we should handle rate here. The StatsD implementation <https://github.com/jsocol/pystatsd/blob/main/statsd/client/base.py#L63> picked a random number and if it was less than rate then it did nothing, I think? > > It looks like this is used as a rate-limiting feature. You provide a rate which must be between 0.00 and 0.99 (basically a percent). Then a random number between 0.00 and 1.00 is generated and if it's larger than the rate value you provided it short circuits and if it's less, then it emits the metric. > So if you provide 0.95 as your rate, most numbers generated by random.random() will be below 0.95 so you'll emit most of the time (i.e. emit 95% of the time). If you provide a rate of 0.05, most numbers generated by random.random() will be above that and so it'll short circuit and not emit the metric (i.e. only emit 5% of the time). > If you provide a value greater than or equal to 1 the whole rate limiting branch is not run (because you're saying you want it emitted 100% of the time essentially) > > — > Reply to this email directly, view it on GitHub <https://github.com/apache/airflow/pull/30873#discussion_r1178403467>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZNLLSM2J7OHTTXHJ5WDE3XDGFRNANCNFSM6AAAAAAXLWBQVI>. > You are receiving this because you were mentioned. > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
