Yes, it is part of V1. I have added it now. I will follow up on the KIP comments in the next couple of days and try to close the review in the next few weeks.
Regards. On Thu, Feb 1, 2024 at 7:40 PM Francois Visconte <francois.visco...@datadoghq.com.invalid> wrote: > > Hi, > > I see that the ticket has been left untouched since a while now. > Should it be included in the tiered storage v1? > We've observed that lacking a way to throttle uploads to tiered storage has > a major impact on > producers and consumers when tiered storage access recovers (starving disk > IOps/throughput or CPU). > For this reason, I think this is an important feature and possibly worth > including in v1? > > Regards, > > > On Tue, Dec 5, 2023 at 8:43 PM Jun Rao <j...@confluent.io.invalid> wrote: > > > Hi, Abhijeet, > > > > Thanks for the KIP. A few comments. > > > > 10. remote.log.manager.write.quota.default: > > 10.1 For other configs, we > > use replica.alter.log.dirs.io.max.bytes.per.second. To be consistent, > > perhaps this can be sth like remote.log.manager.write.max.bytes.per.second. > > 10.2 Could we list the new metrics associated with the new quota. > > 10.3 Is this dynamically configurable? If so, could we document the impact > > to tools like kafka-configs.sh and AdminClient? > > > > Jun > > > > On Tue, Nov 28, 2023 at 2:19 AM Luke Chen <show...@gmail.com> wrote: > > > > > Hi Abhijeet, > > > > > > Thanks for the KIP! > > > This is an important feature for tiered storage. > > > > > > Some comments: > > > 1. Will we introduce new metrics for this tiered storage quotas? > > > This is important because the admin can know the throttling status by > > > checking the metrics while the remote write/read are slow, like the rate > > of > > > uploading/reading byte rate, the throttled time for upload/read... etc. > > > > > > 2. Could you give some examples for the throttling algorithm in the KIP > > to > > > explain it? That will make it much clearer. > > > > > > 3. To solve this problem, we can break down the RLMTask into two smaller > > > tasks - one for segment upload and the other for handling expired > > segments. > > > How do we handle the situation when a segment is still waiting for > > > offloading while this segment is expired and eligible to be deleted? > > > Maybe it'll be easier to not block the RLMTask when quota exceeded, and > > > just check it each time the RLMTask runs? > > > > > > Thank you. > > > Luke > > > > > > On Wed, Nov 22, 2023 at 6:27 PM Abhijeet Kumar < > > abhijeet.cse....@gmail.com > > > > > > > wrote: > > > > > > > Hi All, > > > > > > > > I have created KIP-956 for defining read and write quota for tiered > > > > storage. > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas > > > > > > > > Feedback and suggestions are welcome. > > > > > > > > Regards, > > > > Abhijeet. > > > > > > > > > -- Abhijeet.
