[
https://issues.apache.org/jira/browse/SAMZA-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993697#comment-13993697
]
Chris Riccomini commented on SAMZA-124:
---------------------------------------
YARN's host-level request seem really broken.
The three solutions that I see are:
# Fix YARN
# Keep asking for containers in the AM until we get containers on all hosts,
then round-robin among them, and release the excess ones.
# Use fair share scheduler in the short term if you care about even
distribution of SamzaContainers in YARN.
(1) looks like it will take a fair amount of time. It's the right solution,
though.
(2) Seems even more hacky than just adding anti-affinity into the AM. Now, not
only would we be hacking anti-affinity into the AM, but we'd also be hacking
the way we implement it to accommodate YARN's broken state.
(3) has been tested, works out of the box, and requires no changes to Samza's
code.
For now, I'm proceeding with 3.
> Provide AM based antiaffinity to avoid clumping of containers
> -------------------------------------------------------------
>
> Key: SAMZA-124
> URL: https://issues.apache.org/jira/browse/SAMZA-124
> Project: Samza
> Issue Type: Improvement
> Components: container
> Affects Versions: 0.6.0
> Reporter: Jakob Homan
> Assignee: Chris Riccomini
> Fix For: 0.7.0
>
>
> We're seeing YARN scheduling lots of containers on the same box causing NIC
> saturation and cluster degradation. Pending SAMZA-74 (and its dependency,
> YARN-1042), it would be good to have some basic anti-affinity support at the
> AM level. We can be selective about what nodes we request and accept from
> the RM, hopefully spreading the containers around the cluster more evenly.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
