Hi. I am developing a simple log counting application using Kafka Streams 0.10.1.1. Its implementation is almost the same as the WordCountProcessor in the confluent document [http://docs.confluent.io/3.1.1/streams/developer-guide.html#processor-api]. I am using in-memory state store, its key is the ID of log category, value is count. All the changelogs are written to a broker by context.commit() for fault tolerance, but since the data I handle is large and the size of key is large, it takes a long time to process. Even if it is compacted with a broker, this will put a load on the broker. I would like to write only the latest records for each key on the broker instead of all changelogs at context.commit(). This will reduce the load on the broker and I do not think there will be any negative impact on fault tolerance. If I use the persistent state store, I can do this by enabling caching, but I couldn't find how to accomplish this with the in-memory state store. Can I do this?
Thank you, -- Daisuke
