Re: Random access to kafka messages

Hello, One possible drawback you may want to consider is that the proposed scenario will not work on a DR cluster maintained by MirrorMaker. Other than that it sounds interesting. Best regards, Radu On Fri, Jan 24, 2025 at 12:49 PM Jan Wypych wrote: > Hello, > > We are currently designing a sy

Re: Random access to kafka messages

inal Message- > From: Greg Harris > Sent: Friday, January 24, 2025 9:02 PM > To: Users > Subject: Re: Random access to kafka messages > > Hello Jan, > > I also have not heard of a use case like this for Kafka. One statistic > that I think you might need to manage is

RE: Random access to kafka messages

here) batch sizes can be an issue, but they are configurable on the individual producer and consumer level. Best regards, Jan   -Original Message- From: Greg Harris Sent: Friday, January 24, 2025 9:02 PM To: Users Subject: Re: Random access to kafka messages Hello Jan, I also have

Re: Random access to kafka messages

Date: Saturday, 25 January 2025 at 7:02 am To: Users Subject: Re: Random access to kafka messages EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hello Jan, I also have not heard of a use case like this for Kafka. One statistic that I think you might need to manage is batch

Re: Random access to kafka messages

Hello Jan, I also have not heard of a use case like this for Kafka. One statistic that I think you might need to manage is batch size, and its effect on compression and read amplification. Larger batches on the producer side can make your producers more performant and compression more effective.

Re: Random access to kafka messages

Hi Jan, One immediate concern (which you probably have thought through) is that you will have to be careful with configuring Kafka's cleanup policy. From my understanding, in "delete" cleanup mode, Kafka calculates which messages to delete based on a combination of the message's age and the ratio

Random access to kafka messages

Hello, We are currently designing a system that will ingest some XML messages and then it will store them into some kind of long-term storage (years). The access pattern to data shows that new messages (1-2 weeks old) will be frequent, older data will be accessed rarely. We currently chose Kafk

Re: Random access to kafka messages

Hi, The data you gathered shows promising results, one thing the consider is testing how the Page Cache that Kafka utilizes affect the response times, which greatly improves response time for the fetch requests that are already in the cache since it is stored in memory and may give an impression