MaoYuan Xian created HAMA-759:
---------------------------------
Summary: When peer sends message, not matter what kinds of queue
used, all messages read into memory.
Key: HAMA-759
URL: https://issues.apache.org/jira/browse/HAMA-759
Project: Hama
Issue Type: Improvement
Components: bsp core
Affects Versions: 0.6.1
Reporter: MaoYuan Xian
Before bsp peer sends message to network, all messages have to be read into
memory, No matter what kinds of outgoing queue used. Which means, even when
user use DiskQueue to sending message, no memory saving exactly. In sync()
method of BSPPeerImpl.java，
{code}
while (it.hasNext()) {
...
BSPMessageBundle<M> bundle = combineMessages(messages);
// remove this message during runtime to save a bit of memory
it.remove();
if (combiner != null) {
try {
messenger.transfer(addr, bundle);
} catch (Exception e) {
LOG.error("Error while sending messages", e);
}
{code}
In the combineMessages method, all messages are put into memory:
{code}
private final BSPMessageBundle<M> combineMessages(Iterable<M> messages) {
BSPMessageBundle<M> bundle = new BSPMessageBundle<M>();
if (combiner != null) {
bundle.addMessage(combiner.combine(messages));
} else {
for (M message : messages) {
bundle.addMessage(message);
}
}
return bundle;
}
{code}
Can we make some modifications here, implements a configurable size sending
bundle, only collect parts of message instead of all before sends messages to
network.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
