Re: Are Kafka and Kafka Streams right tools for my case?

2021-01-19 Thread Guozhang Wang
Hello, I have observed several use cases similar to yours that are using Kafka / Kafka Streams in production. That being said, your concerns are valid: * Big messages: 5MB is indeed large, but not extremely big for Kafka. If it is a single message of hundreds of MBs or over a GB then it's a bit

Are Kafka and Kafka Streams right tools for my case?

2021-01-19 Thread The Real Preacher
I'm new to Kafka and will be grateful for any advice We are updating a legacy application together with moving it from IBM MQ to something different. Application currently does the following: * Reads batch XML messages (up to 5 MB) * Parses it to something meaningful * Processes data