Sorry, I’m not sure why the formatting of the email appears to be somewhat disorganized. Therefore, I have reorganized part of the disordered content and added it to doc[1].
Thank you. [1] https://docs.google.com/document/d/1kVa0V_RrPpT5aa5rfxEaH-QxyXTplQr65xUMZMmDoFA/edit?tab=t.0#heading=h.x6o7d75qh2vl Best, Yuepeng On 2025/01/27 02:46:28 Yuepeng Pan wrote: > Thanks Jay Kataria for the comments. > > > > > > 1. Can you give an example of the scenarios where this can be useful. > > > Adding rate limiters to logs seems like an interesting idea, but just > > > wondering what is the business motivation. > > > > > > 3. I am interested in what you talked about - dimensions and allow > > > thresholds to be shared across these dimensions or metrics. Could you give > > > an example of this particularly, I just want to know about the real world > > > applications of this. > > > > > Please let me have a try on clarifing it. > > > > > Generally speaking, the logging rate of each logger varies. > > In some scenarios or under the influence of existing filters, > > if a particular logger generates logs at an especially high rate, > > the log output of other loggers might be affected. > > In short, all loggers compete for the same type of rate-limited resources > without any proactive intervention logic. > > > > > For example, suppose there are logger1 and logger2, > > and the user is interested in the log output of logger2. > > A filter is configured to limit the log rate to 100 records/min. > > If logger1 produces logs at a rate of 200 records/min, > > it is highly likely that logger2 will be unable to output any logs > > because logger1 has already reached the rate-limiting threshold. > > The user expects that while ensuring the rate-limiting of logs, > > the target logger should still be able to output the necessary logs. > > At the least, no logger should be completely blocked from outputting logs due > to rate-limiting. > > > > > The best solution in this case is to set a shared rate-limiting condition for > each logger. > > For example, allow each logger to output 100 records/min. > > This way, every logger is guaranteed a certain log output rate under > rate-limiting. > > When the number of loggers is small, or when the log generation rate of the > process is relatively low, > > even if each logger has reached the rate-limiting threshold, some output can > still be allowed. > > This refers to the shared rate-limited resources or thresholds among all > loggers. > > In this rate limiter, this corresponds to a process-level rate-limiting > threshold. > > > > > I drafted an example to illustrate how loggers can isolate rate-limited > resources and compete for shared rate-limited resources. > > > > > - The filter limiter statistics window is 1min. > > - Filter configs: > > - process level: 1000 records/min > > - logger level: 500 records/min > > - All loggers in the system: logger1, logger2 > > - Statistics of already generated log records > > - Process Stats: 998 records > > - logger1 Stats: 499 records > > - logger2 Stats: 499 records > > - The new log records sequences > > NO_n: record n of logger1 > > Result: Here's remaining 1 record in the current threshold of logger1 (499 to > 500), > > so the record n is allowed to print. > > Stats change: > > - Process Stats: 999 records > > - logger1 Stats: 500 records > > NO_n+1. record n+1 of logger1 > > Result: Here's remaining 0 record in the current threshold of logger1 (500 to > 500). > > but here's remaining 1 record in the process level threshold (999 to 1000). > > So the record n+1 is allowed to print. > > Stats change: > > - Process Stats: 1000 records > > - logger1 Stats: 501 records > > NO_n+2. record n+2 of logger1 > > Result: Here are no remaining records in threshold of logger1 level and > process level. > > So the record n+2 is not allowed to print. > > Stats change: N.A > > NO_n+3. record n+3 of logger2 > > Result: Here's remaining 0 record in the current threshold of process level > (1000 to 1000.) > > but here's remaining 1 record in the logger2 level threshold (499 to 500). > > So the record n+3 is allowed to print. > > Stats change: > > - Process Stats: 1001 records > > - logger2 Stats: 500 records > > > > > NO_...: Subsequent logs will no longer be output as both dedicated > > rate-limited resources and shared rate-limited resources have been exhausted. > > > > > > > > > 2. Number of customers requesting this feature? Maintenance as @Piotr > > > Karwasz <pkarw...@apache.org> , mentioned is going to be a 5 - 10 year > > > period, if we do not have enough customers requesting this, then > > > maintenance of this feature + efforts might not be worth it. > > > > > Thanks for the response. Sorry, I was not aware of this rule before. > > > > > I'm not aware of the actual size of the user group with such needs. > > If necessary, perhaps we could conduct a survey in the user mailing list. > > This email is merely a discussion. If it is prohibited based on this > > rule before the discussion even begins, it might not be a bad thing, > > as it could help everyone avoid unnecessary discussions. > > > > > Best, > Yuepeng > > > > > > > > > > > > At 2025-01-27 03:32:28, "Jay Kataria" <jaykataria1...@gmail.com> wrote: > >Hi Yuepeng, > > > >This seems interesting there are a few comments that I have based on the > >doc and the feature request: > > > >1. Can you give an example of the scenarios where this can be useful. > >Adding rate limiters to logs seems like an interesting idea, but just > >wondering what is the business motivation. > >2. Number of customers requesting this feature? Maintenance as @Piotr > >Karwasz <pkarw...@apache.org> , mentioned is going to be a 5 - 10 year > >period, if we do not have enough customers requesting this, then > >maintenance of this feature + efforts might not be worth it. > >3. I am interested in what you talked about - dimensions and allow > >thresholds to be shared across these dimensions or metrics. Could you give > >an example of this particularly, I just want to know about the real world > >applications of this. > > > > > >Regards, > >Jay Katariya > > > > > > > >On Sun, Jan 26, 2025 at 2:57 AM Yuepeng Pan <panyuep...@apache.org> wrote: > > > >> Hi, community, > >> > >> > >> > >> > >> In some business scenarios, users expect the log rate limit thresholds to > >> be influenced > >> > >> by different dimensions and allow thresholds to be shared across these > >> dimensions or metrics. > >> > >> This enables the system to flexibly output as many logs as possible within > >> the safe constraints of the thresholds. > >> > >> Therefore, it is meaningful to introduce rate limiters based on process > >> granularity and logger granularity, > >> > >> targeting both the number of log entries and the size of the logs. > >> > >> > >> > >> > >> So, I'd like to start a discussion about 'Support a cross-rate Filter > >> based on process and logger granularity'.[1] > >> > >> > >> > >> > >> Looking forward to your attention and comments. > >> > >> > >> > >> > >> Thank you. > >> > >> > >> > >> > >> [1] > >> https://docs.google.com/document/d/1kVa0V_RrPpT5aa5rfxEaH-QxyXTplQr65xUMZMmDoFA/edit?tab=t.0#heading=h.jfuayzme0ome > >> > >> > >> > >> > >> Best, > >> > >> Yuepeng Pan >
