Hi David, Have you tried increasing the number of containers? Our benchmarks show about 1M messages per sec processed on one machine. You should be able to scale linearly by increasing number of containers. All relevant Kafka producer properties like batch size, compression and linger can be passed to Samza.
For concurrency, the upcoming release of Samza includes the multi threading feature. It will provide built in support for processing multiple tasks in parallel. SAMZA-863 and SAMZA-961 are relevant jiras tracking this feature. Thanks On Wednesday, July 27, 2016, David Yu <david...@optimizely.com> wrote: > Is there a way to control the number of producers? Our Samza job writes a > lot of data to the downstream Kafka topic. I was wondering if there is a > way to optimize concurrency by creating more async producers. > > Thanks, > David > -- Sent from my iphone.
