[
https://issues.apache.org/jira/browse/SAMZA-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993816#comment-13993816
]
Zhijie Shen commented on SAMZA-124:
-----------------------------------
[~criccomini], I ran into a similar issue before with a MR job, but not pretty
sure the reason was exactly the same, and I could recall the issue was reported
by users more than once. I think YARN folks want to fix it definitely.
W.R.T 2, It's somehow not hacky given the current scheduler capability. MR is
actually doing something similar. MR even doesn't use AMRMClient. It has it's
own logic to request containers, and see whether the allocated containers are
node local, rack local, not on blacklisted nodes and so on to decide whether to
use the container or given it back to YARN.
W.R.T 3, It would be fine to use share scheduler given the cluster only has
Samza jobs as the only workload.
> Provide AM based antiaffinity to avoid clumping of containers
> -------------------------------------------------------------
>
> Key: SAMZA-124
> URL: https://issues.apache.org/jira/browse/SAMZA-124
> Project: Samza
> Issue Type: Improvement
> Components: container
> Affects Versions: 0.6.0
> Reporter: Jakob Homan
> Assignee: Chris Riccomini
> Fix For: 0.7.0
>
>
> We're seeing YARN scheduling lots of containers on the same box causing NIC
> saturation and cluster degradation. Pending SAMZA-74 (and its dependency,
> YARN-1042), it would be good to have some basic anti-affinity support at the
> AM level. We can be selective about what nodes we request and accept from
> the RM, hopefully spreading the containers around the cluster more evenly.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
