Chris Riccomini created SAMZA-111:
-------------------------------------
Summary: SystemConsumers is slow with large partition count
Key: SAMZA-111
URL: https://issues.apache.org/jira/browse/SAMZA-111
Project: Samza
Issue Type: Bug
Components: container
Affects Versions: 0.6.0
Reporter: Chris Riccomini
We have been seeing very slow processing speed when running a Samza container
that consumes from 1000s of partitions. We don't see a corresponding slow speed
when running the same code, but with fewer input partitions (say 8-24).
The messages per second seems to drop off as more partitions are added to the
Samza container. One Samza job has ~2500 partitions, and is seeing only 6000
messages/sec. The same code running with ~9 partitions is seeing 30,000
messages/sec.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
