Re: log compaction scaling with ~100m messages

Just want to chime on this question as this does seem a good option to avoid some memory hungry K,V store in case we are ok with some async processing. There are cases where you want a combination of some near realtime and some offline processing of the same index and as kafka topic is much

Re: log compaction scaling with ~100m messages

Thank you for your response! Our use case is more similar to a traditional k/v store. We are doing a new process that is going to spit huge amounts of data. We are using kafka as a broker so that downstream clients can all consume from the kafka topic. What we would like to avoid, is writing

Re: log compaction scaling with ~100m messages

Hi, just want to pick this up again. You can always use more partitions to reduce the number of keys handled by a single broker and parallelize the compaction. So with sufficient number of machines and the ability to partition I don’t see you running into problems. Jan On 07.10.2015 05:34,

Re: log compaction scaling with ~100m messages

Using log compaction is well-suited for applications that use Kafka directly and need to persist some state associated with its processing. So something like offset management for consumers is a good fit. Another good use-case is for

log compaction scaling with ~100m messages

hi! We have a use case where we want to store ~100m keys in kafka. Is there any problem with this approach? I have heard from some people using kafka, that kafka has a problem when doing log compaction with those many number of keys. Another topic might have around 10 different K/V pairs for