Hi All: Bumping this thread for more discussion. I’d really appreciate more suggestions on this optional feature for tiered storage. Thanks a lot !
Regards Jian jian fu <[email protected]> 于2025年12月4日周四 21:54写道： > Hi All: > > I updated the KIP content according to Kamal and Haiying's discussion: > 1 Explicitly emphasized that this is a topic-level optional feature > intended for users who prioritize cost. > 1 Added the cost-saving calculation example > 2 Added additional details about the operational drawback of this > feature: need extra disk expansion for the case: long time remote > storage's outage. > 3 Added the scenarios where it may not be very suitable/ beneficial to > enable the feature such as the topic's ratio for remote:local retention is > a very big value. > > Thanks again for joining the discussion. > > Regards > Jian > > jian fu <[email protected]> 于2025年12月2日周二 20:27写道： > >> Hi Kamal: >> >> I think I understand what you mean now. I’ve updated the picture in the >> link(https://github.com/apache/kafka/pull/20913#issuecomment-3601274230) >> . >> Could you help double-check whether we’ve reached the same understanding? >> In short. the drawback of this KIP is that, during a long time remote >> storage outage. it will occupied more disk. The max value is the redundant >> part we saving. >> Thus. After the outage recovered. It will come back to the beginning. >> Pls help to correct me if my understanding is wrong! Thanks again. >> >> Regards >> Jian >> >> Kamal Chandraprakash <[email protected]> 于2025年12月2日周二 >> 19:29写道： >> >>> The already uploaded segments are eligible for deletion from the broker. >>> So, when remote storage is down, >>> then those segments can be deleted as per the local retention settings >>> and >>> new segments can occupy those spaces. >>> This provides more time for the Admin to act when remote storage is down >>> for a longer time. >>> >>> This is from a reliability perspective. >>> >>> On Tue, Dec 2, 2025 at 4:47 PM jian fu <[email protected]> wrote: >>> >>> > Hi Kamal and Haiying Cai: >>> > >>> > maybe you notice that my kafka clusters set 1day local + 3 days-7 days >>> > remote. thus Haiying Cai‘s configure is 3 hours local + 3 days remote. >>> > >>> > I can explain more about my configure. >>> > I try to avoid the latency for some delay consumer to access the >>> remote. >>> > Maybe some applications may encounter some unexpected issue. but we >>> need to >>> > give enough time to handle it. In the period, we don't want the >>> consumer to >>> > access the remote to hurt the whole kafka clusters. So one day is our >>> > expectation. >>> > >>> > I saw one statement in Haiying Cai KIP1248: >>> > " Currently, when a new consumer or a fallen-off consumer requires >>> fetching >>> > messages from a while ago, and those messages are no longer present in >>> the >>> > Kafka broker's local storage, the broker must download the message >>> from the >>> > remote tiered storage and subsequently transfer the data back to the >>> > consumer. " >>> > Extend the local retention time is how we try to avoid the issue >>> (Here, we >>> > don't consider the case one new consumer use the earliest strategy to >>> > consume. it is not often happen in our cases.) >>> > >>> > So. based my configure. I will see there is one day's duplicated >>> segment >>> > wasting in remote storage. Thus I don't use them for real time analyst >>> or >>> > care about the fast reboot or some thing else. So propose this KIP to >>> take >>> > one topic level optional feature to help us to reduce waste and save >>> money. >>> > >>> > Regards >>> > Jian >>> > >>> > jian fu <[email protected]> 于2025年12月2日周二 18:42写道： >>> > >>> > > Hi Kamal: >>> > > >>> > > Thanks for joining this discussion. Let me try to classify my >>> understands >>> > > for your good questions: >>> > > >>> > > 1 Kamal : Do you also have to update the RemoteCopy lag segments and >>> > > bytes metric? >>> > > Jian: The code just delay the upload time for local segment. So >>> it >>> > > seems there is no need to change any lag segments or metrics. right? >>> > > >>> > > 2 Kamal : As Haiying mentioned, the segments get eventually >>> uploaded >>> > to >>> > > remote so not sure about the >>> > > benefit of this proposal. And, remote storage cost is considered as >>> low >>> > > when compared to broker local-disk. >>> > > Jian: The cost benefit is about the total size for occupied. >>> Take >>> > AWS >>> > > S3 as example. Tiered price for: 1 GB is 0.02 USD (You can refer to >>> > > https://calculator.aws/#/createCalculator/S3). >>> > > It is cheaper than local disk. So as I mentioned that the saving >>> money >>> > > depend on the ratio local vs remote retention time. If your set the >>> > remote >>> > > storage time as a long time. The benefit is few, It is just avoiding >>> the >>> > > waste instead of cost saving. >>> > > So I take it as topic level optional configure instead of default >>> > > feature. >>> > > >>> > > 3 Kamal: It provides some cushion during third-party object >>> storage >>> > > downtime. >>> > > Jian: I draw one picture to try to under the logic( >>> > > https://github.com/apache/kafka/pull/20913#issuecomment-3601274230). >>> You >>> > > can help to check if my understanding is right. I seemed that no >>> > difference >>> > > for them. So for this question. maybe we need to discuss more about >>> it. >>> > The >>> > > only difference maybe we may increase a little local disk for temp >>> due to >>> > > the delay for upload remote. So in the original proposal. I want to >>> > upload >>> > > N-1 segments. But it seems the value is not much. >>> > > >>> > > BTW. I want to classify one basic rule: this feature isn't to change >>> the >>> > > default behavior. and the saving amount is not very big value in all >>> > cases. >>> > > It is suitable for part of topic which set a low ratio for >>> remote/local >>> > > such as 7days/1days or 3days/1day >>> > > At the last. Thanks again for your time and your comments. All the >>> > > questions are valid and good for us to thing more about it. >>> > > >>> > > Regards >>> > > Jian >>> > > >>> > > >>> > > Kamal Chandraprakash <[email protected]> 于2025年12月2日周二 >>> > > 17:41写道： >>> > > >>> > >> 1. Do you also have to update the RemoteCopy lag segments and bytes >>> > >> metric? >>> > >> 2. As Haiying mentioned, the segments get eventually uploaded to >>> remote >>> > so >>> > >> not sure about the >>> > >> benefit of this proposal. And, remote storage cost is considered as >>> low >>> > >> when compared to broker local-disk. >>> > >> It provides some cushion during third-party object storage downtime. >>> > >> >>> > >> >>> > >> On Tue, Dec 2, 2025 at 2:45 PM Kamal Chandraprakash < >>> > >> [email protected]> wrote: >>> > >> >>> > >> > Hi Jian, >>> > >> > >>> > >> > Thanks for the KIP! >>> > >> > >>> > >> > When remote storage is unavailable for a few hrs, then with lazy >>> > upload >>> > >> > there is a risk of the broker disk getting full soon. >>> > >> > The Admin has to configure the local retention configs properly. >>> With >>> > >> > eager upload, the disk utilization won't grow >>> > >> > until the local retention time (expectation is that all the >>> > >> > passive segments are uploaded). And, provides some time >>> > >> > for the Admin to take any action based on the situation. >>> > >> > >>> > >> > -- >>> > >> > Kamal >>> > >> > >>> > >> > On Tue, Dec 2, 2025 at 10:28 AM Haiying Cai via dev < >>> > >> [email protected]> >>> > >> > wrote: >>> > >> > >>> > >> >> Jian, >>> > >> >> >>> > >> >> Understands this is an optional feature and the cost saving >>> depends >>> > on >>> > >> >> the ratio between local.retention.ms and total retention.ms. >>> > >> >> >>> > >> >> In our setup, we have local.retention set to 3 hours and total >>> > >> retention >>> > >> >> set to 3 days, so the saving is not going to be significant. >>> > >> >> >>> > >> >> On 2025/12/01 05:33:11 jian fu wrote: >>> > >> >> > Hi Haiying Cai, >>> > >> >> > >>> > >> >> > Thanks for joining the discussion for this KIP. All of your >>> > concerns >>> > >> are >>> > >> >> > valid, and that is exactly why I introduced a topic-level >>> > >> configuration >>> > >> >> to >>> > >> >> > make this feature optional. This means that, by default, the >>> > behavior >>> > >> >> > remains unchanged. Only when users are not pursuing faster >>> broker >>> > >> boot >>> > >> >> time >>> > >> >> > or other optimizations — and care more about cost — would they >>> > enable >>> > >> >> this >>> > >> >> > option to some topics to save resources. >>> > >> >> > >>> > >> >> > Regarding cost self: the actual savings depend on the ratio >>> between >>> > >> >> local >>> > >> >> > retention and remote retention. In the KIP/PR, I provided a >>> test >>> > >> >> example: >>> > >> >> > if we configure 1 day of local retention and 2 days of remote >>> > >> >> retention, we >>> > >> >> > can save about 50%. And realistically, I don't think anyone >>> would >>> > >> boldly >>> > >> >> > set local retention to a very small value (such as minutes) >>> due to >>> > >> the >>> > >> >> > latency concerns associated with remote storage. So in short, >>> the >>> > >> >> feature >>> > >> >> > will help reduce cost, and the amount saved simply depends on >>> the >>> > >> ratio. >>> > >> >> > Take my company's usage as real example, we configure most of >>> the >>> > >> >> topics: 1 >>> > >> >> > day of local retention and 3–7 days of remote storage (3 days >>> for >>> > >> topic >>> > >> >> > with log/metric usage, 7 days for topic with normal business >>> > usage). >>> > >> >> and we >>> > >> >> > don't care about the boot speed and some thing else, This KIP >>> > allows >>> > >> us >>> > >> >> to >>> > >> >> > save 1/7 to 1/3 of the total disk usage for remote storage. >>> > >> >> > >>> > >> >> > Anyway, this is just a topic-level optional feature which don't >>> > >> reject >>> > >> >> the >>> > >> >> > benifit for current design. Thanks again for the discussion. I >>> can >>> > >> >> update >>> > >> >> > the KIP to better classify scenarios where this optional >>> feature is >>> > >> not >>> > >> >> > suitable. Currently, I only listed real-time analytics as the >>> > >> negative >>> > >> >> > example. >>> > >> >> > >>> > >> >> > Welcome further discussion to help make this KIP more complete. >>> > >> Thanks! >>> > >> >> > >>> > >> >> > Regards, >>> > >> >> > Jian >>> > >> >> > >>> > >> >> > Haiying Cai via dev <[email protected]> 于2025年12月1日周一 >>> > 12:40写道： >>> > >> >> > >>> > >> >> > > Jian, >>> > >> >> > > >>> > >> >> > > Thanks for the contribution. But I feel the uploading the >>> local >>> > >> >> segment >>> > >> >> > > file to remote storage ASAP is advantageous in several >>> scenarios: >>> > >> >> > > >>> > >> >> > > 1. Enable the fast bootstrapping a new broker. A new broker >>> > >> doesn’t >>> > >> >> have >>> > >> >> > > to replicate all the data from the leader broker, it only >>> needs >>> > to >>> > >> >> > > replicate the data from the tail of the remote log segment >>> to the >>> > >> >> tail of >>> > >> >> > > the current end of the topic (LSO) since all the other data >>> are >>> > in >>> > >> the >>> > >> >> > > remote tiered storage and it can download them later lazily, >>> this >>> > >> is >>> > >> >> what >>> > >> >> > > KIP-1023 trying to solve; >>> > >> >> > > 2. Although nobody has proposed a KIP to allow a consumer >>> client >>> > to >>> > >> >> read >>> > >> >> > > from the remote tiered storage directly, but this will helps >>> the >>> > >> >> > > fall-behind consumer to do catch-up reads or perform the >>> > backfill. >>> > >> >> This >>> > >> >> > > path allows the consumer backfill to finish without >>> polluting the >>> > >> >> broker’s >>> > >> >> > > page cache. The earlier the data is on the remote tiered >>> > storage, >>> > >> >> the more >>> > >> >> > > advantageous it is for the client. >>> > >> >> > > >>> > >> >> > > I think in your Proposal, you are delaying uploading the >>> segment >>> > >> but >>> > >> >> the >>> > >> >> > > file will still be uploaded at a later time, I guess this can >>> > >> saves a >>> > >> >> few >>> > >> >> > > hours storage cost for that file in the remote storage, not >>> sure >>> > >> >> whether >>> > >> >> > > that is a significant cost saved (if the file needs to stay >>> in >>> > >> remote >>> > >> >> > > tiered storage for several days or weeks due to retention >>> > policy). >>> > >> >> > > >>> > >> >> > > On 2025/11/19 13:29:11 jian fu wrote: >>> > >> >> > > > Hi everyone, I'd like to start a discussion on KIP-1241, >>> the >>> > goal >>> > >> >> is to >>> > >> >> > > > reduce the remote storage. KIP: >>> > >> >> > > > >>> > >> >> > > >>> > >> >> >>> > >> >>> > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1241%3A+Reduce+tiered+storage+redundancy+with+delayed+upload >>> > >> >> > > > >>> > >> >> > > > The Draft PR: https://github.com/apache/kafka/pull/20913 >>> > >> >> Problem: >>> > >> >> > > > Currently, >>> > >> >> > > > Kafka's tiered storage implementation uploads all >>> non-active >>> > >> local >>> > >> >> log >>> > >> >> > > > segments to remote storage immediately, even when they are >>> > still >>> > >> >> within >>> > >> >> > > the >>> > >> >> > > > local retention period. >>> > >> >> > > > This results in redundant storage of the same data in both >>> > local >>> > >> and >>> > >> >> > > remote >>> > >> >> > > > tiers. >>> > >> >> > > > >>> > >> >> > > > When there is no requirement for real-time analytics or >>> > immediate >>> > >> >> > > > consumption based on remote storage. It has the following >>> > >> drawbacks: >>> > >> >> > > > >>> > >> >> > > > 1. Wastes storage capacity and costs: The same data is >>> stored >>> > >> twice >>> > >> >> > > during >>> > >> >> > > > the local retention window >>> > >> >> > > > 2. Provides no immediate benefit: During the local >>> retention >>> > >> period, >>> > >> >> > > reads >>> > >> >> > > > prioritize local data, making the remote copy unnecessary >>> > >> >> > > > >>> > >> >> > > > >>> > >> >> > > > So. this KIP is to reduce tiered storage redundancy with >>> > delayed >>> > >> >> upload. >>> > >> >> > > > You can check the test result example here directly: >>> > >> >> > > > >>> > >> https://github.com/apache/kafka/pull/20913#issuecomment-3547156286 >>> > >> >> > > > Looking forward to your feedback! Best regards, Jian >>> > >> >> > > > >>> > >> >> > >>> > >> > >>> > >> > >>> > >> >>> > > >>> > > >>> > > >>> > > >>> > >>> >> >> >> >> >> > > > -- Regards Fu.Jian
