I would like to use Kafka to process messages that need to be immutably stored for a N-days, and during that period the msgs need to be indexed, searched, as well as retrieval of msg data that is queried.
One approach is to read messages from Kafka and store the messages in a secondary db for query and data retrieval. Once the messages are read and processed into the secondary db, then the messages can be discarded from the Kafka queue. Another approach is to read the messages, build an external index for searching that directly references the message data by Kafka-key in the Kafka queue itself. In this case the Kafka becomes the message store for the life of the message/data. The latter would be ideal for me if the performance of query-by-key and message data retrieval is very good. Is random query of message+data good for Kafka? Is this an appropriate usecase for Kafka? Thank you. Marko. .
