There are two ways you can configure your topics, log compaction and with no cleaning. The choice depends on your use case. Are the records uniquely identifiable and will they receive updates? Then log compaction is the way to go. If they are truly read only, you can go without log compaction.
We have a small processes which consume a topic and perform upserts to our various database engines. It's easy to change how it all works and simply consume the single source of truth again. I've written a bit about log compaction here: http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/ On Fri, Jul 10, 2015 at 3:46 AM, Daniel Schierbeck < daniel.schierb...@gmail.com> wrote: > I'd like to use Kafka as a persistent store – sort of as an alternative to > HDFS. The idea is that I'd load the data into various other systems in > order to solve specific needs such as full-text search, analytics, indexing > by various attributes, etc. I'd like to keep a single source of truth, > however. > > I'm struggling a bit to understand how I can configure a topic to retain > messages indefinitely. I want to make sure that my data isn't deleted. Is > there a guide to configuring Kafka like this? >
