[
https://issues.apache.org/jira/browse/HAMA-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925659#comment-13925659
]
Anastasis Andronidis commented on HAMA-629:
-------------------------------------------
Yes, I agree.
I was also keep in mind that we do a local aggregation in each peer. My commend
is more on really large deployment (e.g. 1500 peers) with heavy use of the
aggregation framework (heavy use = large chunks of data in the aggregation
messages).
> Improve RPC Scalability Part 2
> ------------------------------
>
> Key: HAMA-629
> URL: https://issues.apache.org/jira/browse/HAMA-629
> Project: Hama
> Issue Type: Sub-task
> Components: graph
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Fix For: 0.7.0
>
>
> There is a problem when all 1k peers would attempt to send to a single peer
> (let's say a master task in a graph algorithm that aggregates). In this case
> the peer will start 1k-threads which is using enourmous amount of memory.
> I think we can coordinate the message sending either with Zookeeper or by
> using the task id and do a smarter sending chain.
> By the last, I mean, that each task can start at a different offset in the
> peer array to start sending messages to the other peers. But this won't solve
> the problem DDoS'ing a single master task.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
