Claudio Martella updated GIRAPH-45:
This a premature patch not meant for inclusion but as RFC. It passes all local
unit tests and MR except checkpointing and partitioner tests.
Apparently I broke something with partitioning.
In case of checkpointing it breaks in
BasicRPCCommunications#checkForMessageToNonExistentVertex(), with messages sent
to the wrong worker (see IllegalStateException), while in TestGraphPartitioner
the output partition files are small than required size.
I'm requesting some comments as apparently I don't get how I broke partitioner
package by moving some code from prepareSuperstep() to putMsg* methods. There
must be an assumption I don't get which might be obvious to one of you.
I tried to go incrementally by just refactoring
BasicRPCCommunications#checkForMessageToNonExistentVertex() and leaving the
rest AS-IS, so no out-of-core classes, just really trunk with
BasicRPCCommunications#checkForMessageToNonExistentVertex() logics, and the
code doesn't break. So... any ideas?
> Improve the way to keep outgoing messages
> Key: GIRAPH-45
> URL: https://issues.apache.org/jira/browse/GIRAPH-45
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Hyunsik Choi
> Attachments: GIRAPH-45.diff
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a
> potential problem to cause out of memory when the rate of message generation
> is higher than the rate of message flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or
> some approach to spill messages into disk.
> The below link is Dmitriy's suggestion.
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
For more information on JIRA, see: http://www.atlassian.com/software/jira