Hello Yuepeng,
Thanks so much for reaching out to us. Your use case is indeed an
interesting one and it is good to learn such Log4j deployments in the wild.
Consider the following Log4j filter pseudo code:
WeakHashMap<Key, RateLimiter> rateLimiterByKey =
activeLoggerContext.getObject("rateLimiters");
Key key = Key.fromDimensions(logEvent.getLogger(), ...);
RateLimiter rateLimiter = rateLimiterByKey.putIfAbsent(key, ignored ->
RateLimiter.ofMaxRate(key.maxRate()));
return rateLimiter.acquire() ? Result.ACCEPT : Result.DENY;
You can either implement this in a Java/Kotlin/Scala/etc. class
<https://logging.apache.org/log4j/2.x/manual/filters.html#extending>
or a Script
Filter <https://logging.apache.org/log4j/2.x/manual/filters.html#Script>.
Would you mind explaining to us why these are not an option for you but
instead this logic must be provided as an official Log4j component, please?
Kind regards.
On Mon, Jan 27, 2025 at 3:55 AM Yuepeng Pan <panyuep...@apache.org> wrote:
> Sorry, I’m not sure why the formatting of the email appears to be somewhat
> disorganized. Therefore, I have reorganized part of the disordered content
> and added it to doc[1].
>
> Thank you.
>
> [1]
> https://docs.google.com/document/d/1kVa0V_RrPpT5aa5rfxEaH-QxyXTplQr65xUMZMmDoFA/edit?tab=t.0#heading=h.x6o7d75qh2vl
>
> Best,
> Yuepeng
>
> On 2025/01/27 02:46:28 Yuepeng Pan wrote:
> > Thanks Jay Kataria for the comments.
> >
> >
> >
> >
> > > 1. Can you give an example of the scenarios where this can be useful.
> >
> > > Adding rate limiters to logs seems like an interesting idea, but just
> >
> > > wondering what is the business motivation.
> >
> >
> >
> >
> > > 3. I am interested in what you talked about - dimensions and allow
> >
> > > thresholds to be shared across these dimensions or metrics. Could you
> give
> >
> > > an example of this particularly, I just want to know about the real
> world
> >
> > > applications of this.
> >
> >
> >
> >
> > Please let me have a try on clarifing it.
> >
> >
> >
> >
> > Generally speaking, the logging rate of each logger varies.
> >
> > In some scenarios or under the influence of existing filters,
> >
> > if a particular logger generates logs at an especially high rate,
> >
> > the log output of other loggers might be affected.
> >
> > In short, all loggers compete for the same type of rate-limited
> resources without any proactive intervention logic.
> >
> >
> >
> >
> > For example, suppose there are logger1 and logger2,
> >
> > and the user is interested in the log output of logger2.
> >
> > A filter is configured to limit the log rate to 100 records/min.
> >
> > If logger1 produces logs at a rate of 200 records/min,
> >
> > it is highly likely that logger2 will be unable to output any logs
> >
> > because logger1 has already reached the rate-limiting threshold.
> >
> > The user expects that while ensuring the rate-limiting of logs,
> >
> > the target logger should still be able to output the necessary logs.
> >
> > At the least, no logger should be completely blocked from outputting
> logs due to rate-limiting.
> >
> >
> >
> >
> > The best solution in this case is to set a shared rate-limiting
> condition for each logger.
> >
> > For example, allow each logger to output 100 records/min.
> >
> > This way, every logger is guaranteed a certain log output rate under
> rate-limiting.
> >
> > When the number of loggers is small, or when the log generation rate of
> the process is relatively low,
> >
> > even if each logger has reached the rate-limiting threshold, some output
> can still be allowed.
> >
> > This refers to the shared rate-limited resources or thresholds among all
> loggers.
> >
> > In this rate limiter, this corresponds to a process-level rate-limiting
> threshold.
> >
> >
> >
> >
> > I drafted an example to illustrate how loggers can isolate rate-limited
> resources and compete for shared rate-limited resources.
> >
> >
> >
> >
> > - The filter limiter statistics window is 1min.
> >
> > - Filter configs:
> >
> > - process level: 1000 records/min
> >
> > - logger level: 500 records/min
> >
> > - All loggers in the system: logger1, logger2
> >
> > - Statistics of already generated log records
> >
> > - Process Stats: 998 records
> >
> > - logger1 Stats: 499 records
> >
> > - logger2 Stats: 499 records
> >
> > - The new log records sequences
> >
> > NO_n: record n of logger1
> >
> > Result: Here's remaining 1 record in the current threshold of logger1
> (499 to 500),
> >
> > so the record n is allowed to print.
> >
> > Stats change:
> >
> > - Process Stats: 999 records
> >
> > - logger1 Stats: 500 records
> >
> > NO_n+1. record n+1 of logger1
> >
> > Result: Here's remaining 0 record in the current threshold of logger1
> (500 to 500).
> >
> > but here's remaining 1 record in the process level threshold (999 to
> 1000).
> >
> > So the record n+1 is allowed to print.
> >
> > Stats change:
> >
> > - Process Stats: 1000 records
> >
> > - logger1 Stats: 501 records
> >
> > NO_n+2. record n+2 of logger1
> >
> > Result: Here are no remaining records in threshold of logger1 level and
> process level.
> >
> > So the record n+2 is not allowed to print.
> >
> > Stats change: N.A
> >
> > NO_n+3. record n+3 of logger2
> >
> > Result: Here's remaining 0 record in the current threshold of process
> level (1000 to 1000.)
> >
> > but here's remaining 1 record in the logger2 level threshold (499 to
> 500).
> >
> > So the record n+3 is allowed to print.
> >
> > Stats change:
> >
> > - Process Stats: 1001 records
> >
> > - logger2 Stats: 500 records
> >
> >
> >
> >
> > NO_...: Subsequent logs will no longer be output as both dedicated
> >
> > rate-limited resources and shared rate-limited resources have been
> exhausted.
> >
> >
> >
> >
> >
> >
> >
> > > 2. Number of customers requesting this feature? Maintenance as @Piotr
> >
> > > Karwasz <pkarw...@apache.org> , mentioned is going to be a 5 - 10 year
> >
> > > period, if we do not have enough customers requesting this, then
> >
> > > maintenance of this feature + efforts might not be worth it.
> >
> >
> >
> >
> > Thanks for the response. Sorry, I was not aware of this rule before.
> >
> >
> >
> >
> > I'm not aware of the actual size of the user group with such needs.
> >
> > If necessary, perhaps we could conduct a survey in the user mailing list.
> >
> > This email is merely a discussion. If it is prohibited based on this
> >
> > rule before the discussion even begins, it might not be a bad thing,
> >
> > as it could help everyone avoid unnecessary discussions.
> >
> >
> >
> >
> > Best,
> > Yuepeng
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2025-01-27 03:32:28, "Jay Kataria" <jaykataria1...@gmail.com> wrote:
> > >Hi Yuepeng,
> > >
> > >This seems interesting there are a few comments that I have based on the
> > >doc and the feature request:
> > >
> > >1. Can you give an example of the scenarios where this can be useful.
> > >Adding rate limiters to logs seems like an interesting idea, but just
> > >wondering what is the business motivation.
> > >2. Number of customers requesting this feature? Maintenance as @Piotr
> > >Karwasz <pkarw...@apache.org> , mentioned is going to be a 5 - 10 year
> > >period, if we do not have enough customers requesting this, then
> > >maintenance of this feature + efforts might not be worth it.
> > >3. I am interested in what you talked about - dimensions and allow
> > >thresholds to be shared across these dimensions or metrics. Could you
> give
> > >an example of this particularly, I just want to know about the real
> world
> > >applications of this.
> > >
> > >
> > >Regards,
> > >Jay Katariya
> > >
> > >
> > >
> > >On Sun, Jan 26, 2025 at 2:57 AM Yuepeng Pan <panyuep...@apache.org>
> wrote:
> > >
> > >> Hi, community,
> > >>
> > >>
> > >>
> > >>
> > >> In some business scenarios, users expect the log rate limit
> thresholds to
> > >> be influenced
> > >>
> > >> by different dimensions and allow thresholds to be shared across these
> > >> dimensions or metrics.
> > >>
> > >> This enables the system to flexibly output as many logs as possible
> within
> > >> the safe constraints of the thresholds.
> > >>
> > >> Therefore, it is meaningful to introduce rate limiters based on
> process
> > >> granularity and logger granularity,
> > >>
> > >> targeting both the number of log entries and the size of the logs.
> > >>
> > >>
> > >>
> > >>
> > >> So, I'd like to start a discussion about 'Support a cross-rate Filter
> > >> based on process and logger granularity'.[1]
> > >>
> > >>
> > >>
> > >>
> > >> Looking forward to your attention and comments.
> > >>
> > >>
> > >>
> > >>
> > >> Thank you.
> > >>
> > >>
> > >>
> > >>
> > >> [1]
> > >>
> https://docs.google.com/document/d/1kVa0V_RrPpT5aa5rfxEaH-QxyXTplQr65xUMZMmDoFA/edit?tab=t.0#heading=h.jfuayzme0ome
> > >>
> > >>
> > >>
> > >>
> > >> Best,
> > >>
> > >> Yuepeng Pan
> >
>
