[
https://issues.apache.org/jira/browse/SAMZA-111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Riccomini updated SAMZA-111:
----------------------------------
Attachment: samza-perf-hacks.0.diff
I did some more investigation into this issue. I'm going to do two things in
this ticket:
1. Fix the SystemStreamConsumers.poll issue where we're polling all streams,
regardless of whether their fetchMap count is 0 or not.
2. Add a MockSystem to do further benchmarking.
I'm then going to open tickets for the numerous other findings I've had while
exploring this issue. Some things to explore:
1. GrizzledSLF4J logging seems to be kind of slow.
2. We seem to be spending a fair chunk of time (~50%) in commit, window, and
send, which are all NOOP operations.
3. Switching RoundRobinChooser from a Scala Set to as Java HashSet seemed to
improve the chooser's performance pretty dramatically.
4. Eliminating all for loops in SystemConsumers.poll, and caching per-system
fetch maps dramatically sped up performance.
5. Scala data structures seem slow in certain spots. Time is being sent in
plus$ and minus$ methods for sets and maps. Switching to Java-based data
structures seemed to fix it.
6. It seems like Scala's for loops were behaving slowly in SamzaContainer, and
SystemConsumers. I was seeing a non-trivial amount of time being spent in
hasNext, and Scala iterators.
I've posted an RB with a bunch of performance hacks I've made.
https://reviews.apache.org/r/16426
With these hacks, I'm able to process at about 230,000 messages/sec with 1000
streams, 12 threads, 4 partitions each.
> SystemConsumers is slow with large partition count
> --------------------------------------------------
>
> Key: SAMZA-111
> URL: https://issues.apache.org/jira/browse/SAMZA-111
> Project: Samza
> Issue Type: Bug
> Components: container
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
> Assignee: Chris Riccomini
> Attachments:
> 12-threads-1000-streams-4-partitions-each-with-hacky-fix.png,
> 12-threads-1000-streams-4-partitions-each.png,
> 12-threads-8-streams-4-partitions-each.png, samza-perf-hacks.0.diff,
> samza-perf-hacks.png
>
>
> We have been seeing very slow processing speed when running a Samza container
> that consumes from 1000s of partitions. We don't see a corresponding slow
> speed when running the same code, but with fewer input partitions (say 8-24).
> The messages per second seems to drop off as more partitions are added to the
> Samza container. One Samza job has ~2500 partitions, and is seeing only 6000
> messages/sec. The same code running with ~9 partitions is seeing 30,000
> messages/sec.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
