[
https://issues.apache.org/jira/browse/GIRAPH-250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Reisman updated GIRAPH-250:
-------------------------------
Attachment: GIRAPH-250-3.patch
Found a small potential loophole in the case that the Math.abs(hashCode()) part
of this comes up Integer.MIN_VALUE, fixed it. Probably not an issue, but better
safe than sorry.
As for port # etc. I looked into it, but the distribution is very good in the
first loop, and anyone who waits or gets a 2nd input split, the "available
split" list has gotten so small by then that they would all hash to similar
indexes anyway. But thats an excellent idea, Jakob mentioned it too.
> Let workers contending for InputSplits during INPUT_SUPERSTEP guess better,
> choose quicker.
> -------------------------------------------------------------------------------------------
>
> Key: GIRAPH-250
> URL: https://issues.apache.org/jira/browse/GIRAPH-250
> Project: Giraph
> Issue Type: Improvement
> Components: bsp, graph, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-250-1.patch, GIRAPH-250-2.patch,
> GIRAPH-250-3.patch
>
>
> In the job logs it has become clear that workers trying to scan for
> master-created Znodes indicating an InputSplit is available to claim (and
> read) are starting very similar lists of znode names to scan (iterating from
> 0 through the list all at the same time)
> what you see in the logs is lots of misses, followed by finally a hit
> somewhere. By using iterating the list, but starting from a different spot
> for each worker (see the patch its a simple change using the hash code of the
> worker hostname + index and mod that by the size of the list of possible
> splits to claim) we (mostly) iterate starting from different parts of the
> input split list each worker gets, thereby lowering contention dramatically
> and ensuring everyone will more quickly claim (at least their first) input
> split. This seems to work very well so far.
> passes mvn verify etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
