Hey folks, I shared this with Omnia offline: One concern I have is with the length of time we keep "seen" producer IDs. It seems like the default is 1 hour. If a producer produces every 2 hours or so, and we are hitting the limit, it seems like we will throttle it even though we've seen it before and have state for it on the server. Then, it seems like we will have to wait for the natural expiration of producer ids (via producer.id.expiration.ms) before we allow new or idle producers to join again without throttling. I think this proposal is a step in the right direction when it comes to throttling the "right" clients, but I want to make sure we have reasonable defaults. Keep in mind that idempotent producers are the default, so most folks won't be tuning these values out of the box.
As for Igor's questions about InitProducerId -- I think the main reason we have avoided that solution is that there is no state stored for idempotent producers when grabbing an ID. My concern there is either storing too much state to track this or throttling before we need to. Justine On Thu, May 2, 2024 at 2:36 PM Claude Warren, Jr <claude.war...@aiven.io.invalid> wrote: > There is some question about whether or not we need the configuration > options. My take on them is as follows: > > producer.id.quota.window.num No opinion. I don't know what this is used > for, but I suspect that there is a good reason to have it. It is not used > within the Bloom filter caching mechanism > producer.id.quota.window.size.seconds Leave it as it is one of the most > effective ways to tune the filter and determines how long a PID is > recognized. > producer.id.quota.cache.cleanup.scheduler.interval.ms Remove it unless > there is another use for it. We can get a better calculation for > internals. > producer.id.quota.cache.layer.count Leave it as it is one of the most > effective ways to tune the filter. > producer.id.quota.cache.false.positive.rate Replace it with a constant, I > don't think any other Bloom filter solution provides access to this knob > for end users. > producer_ids_rate Leave this one, it is critical for reasonable operation. >
