hi!
We have a use case where we want to store ~100m keys in kafka. Is there any
problem with this approach?
I have heard from some people using kafka, that kafka has a problem when doing
log compaction with those many number of keys.
Another topic might have around 10 different K/V pairs for each key in the
primary topic. The primary topic's keyspace is approx of 100m keys. We would
like to store this in kafka because we are doing a lot of stream processing on
these messages, and want to avoid writing another process to recompute data
from snapshots.
So, in summary:
primary topic: ~100m keyssecondary topic: ~1B keys
Is it feasible to use log compaction at such a scale of data?
Thanks
feroze.
