> On 10. jul. 2015, at 15.16, Shayne S <shaynest...@gmail.com> wrote: > > There are two ways you can configure your topics, log compaction and with > no cleaning. The choice depends on your use case. Are the records uniquely > identifiable and will they receive updates? Then log compaction is the way > to go. If they are truly read only, you can go without log compaction.
I'd rather be free to use the key for partitioning, and the records are immutable — they're event records — so disabling compaction altogether would be preferable. How is that accomplished? > > We have a small processes which consume a topic and perform upserts to our > various database engines. It's easy to change how it all works and simply > consume the single source of truth again. > > I've written a bit about log compaction here: > http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/ > > On Fri, Jul 10, 2015 at 3:46 AM, Daniel Schierbeck < > daniel.schierb...@gmail.com> wrote: > >> I'd like to use Kafka as a persistent store – sort of as an alternative to >> HDFS. The idea is that I'd load the data into various other systems in >> order to solve specific needs such as full-text search, analytics, indexing >> by various attributes, etc. I'd like to keep a single source of truth, >> however. >> >> I'm struggling a bit to understand how I can configure a topic to retain >> messages indefinitely. I want to make sure that my data isn't deleted. Is >> there a guide to configuring Kafka like this? >>
