[
https://issues.apache.org/jira/browse/GIRAPH-389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486424#comment-13486424
]
Eli Reisman commented on GIRAPH-389:
------------------------------------
Last time I instrumented that part of the code on cluster runs, I found that
with lots of workers, each claim attempt starts to slow down the syncing of the
ZK quorum because those claims don't just read, they try to write. This forces
a sync(). Our quorum was not very large, and even then the response time of the
quorum to reads and writes would drag to a crawl during this phase of input
reads. These were all problems that start small and get much worse as you scale
out to more workers.
Perhaps the speed increase from multithreading is that they simply get a split
quickly and the whole list is covered, very few 2nd iterations on the split
list are ever needed? Has anyone run big jobs with lots of machines using teh
multithreaded input split phase yet? does it speed up the load-in? Is there a
scale worker-wise where the ZK starts to bog down?
Come to think of it, if each thread group on a worker gets the same split list
ordering from the input split organizer, we might want to modify it to help:
the split organizer tries to keep workers from iterating on identical orderings
of the split list. This eliminated a "mirroring" behavior where groups of
workers would iterate the list from the same start index, and they would slow
down the ZK response time for everyone by making it sync competing claims for
the same splits all the time. The hashed index distributed the workers so that
they don't tend to compete as often for the same split.
Anyway: the hashing is done with host:port so all the locality blocks and the
rest of the original split list will be in identical order for all threads on
the same worker, and the competition is back on again, especially (I bet) when
you get into that nice 1200-1600 worker range where some use cases happen.
For a fix: each thread would need its own "variation" of the split list. the
inplit split organizer could be equipped to generate iterators that build
variations on the current split list it generates. This could be done by: 1.
shuffle() on the locality block before inserting it back into the full split
list and 2. use threadId + host:port for the hash key to make sure they iterate
on different parts of the list if they don't get a local block.
> Multithreading should intelligently allocate the thread pools
> -------------------------------------------------------------
>
> Key: GIRAPH-389
> URL: https://issues.apache.org/jira/browse/GIRAPH-389
> Project: Giraph
> Issue Type: Bug
> Reporter: Avery Ching
> Assignee: Avery Ching
> Fix For: 0.2.0
>
> Attachments: GIRAPH-389.patch
>
>
> Even if the user suggests a very high number of threads, the input split
> threads should not exceed the number splits divided by the number of workers.
> The number of compute threads should not be greater than the number of
> partitions on that worker.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
