[
https://issues.apache.org/jira/browse/SAMZA-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984678#comment-13984678
]
Chris Riccomini commented on SAMZA-124:
---------------------------------------
Patched AM to force container requests to specific hosts. YARN does not appear
to honor these requests.
Here are the hosts that I use in the ContainerRequest:
{noformat}
List(app859, app854, app872)
List(app867, app874, app865)
List(app857, app859, app852)
List(app874, app859, app880)
List(app865, app867, app873)
List(app857, app874, app872)
List(app880, app852, app864)
List(app865, app875, app864)
List(app867, app864, app858)
List(app857, app880, app858)
List(app872, app859, app857)
List(app867, app875, app854)
List(app871, app857, app858)
List(app875, app864, app858)
List(app865, app857, app864)
List(app857, app880, app867)
List(app873, app865, app852)
List(app872, app880, app858)
List(app852, app865, app857)
List(app852, app865, app873)
List(app873, app858, app875)
List(app867, app874, app880)
List(app880, app875, app867)
List(app873, app867, app871)
List(app867, app880, app858)
List(app865, app852, app858)
List(app852, app854, app858)
List(app871, app873, app852)
List(app880, app875, app857)
List(app873, app859, app872)
List(app872, app880, app864)
List(app871, app875, app852)
{noformat}
I use 3 per request, just in case the first one is full.
The resulting distribution is:
{noformat}
14 containers on app854
13 containers on app859
6 containers on app852
{noformat}
Correction on prior comment: we're using the FifoScheduler right now, an don't
have node locality set, since it's capacity-specific (or at least it appears to
be).
> Provide AM based antiaffinity to avoid clumping of containers
> -------------------------------------------------------------
>
> Key: SAMZA-124
> URL: https://issues.apache.org/jira/browse/SAMZA-124
> Project: Samza
> Issue Type: Improvement
> Components: container
> Affects Versions: 0.6.0
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Fix For: 0.7.0
>
>
> We're seeing YARN scheduling lots of containers on the same box causing NIC
> saturation and cluster degradation. Pending SAMZA-74 (and its dependency,
> YARN-1042), it would be good to have some basic anti-affinity support at the
> AM level. We can be selective about what nodes we request and accept from
> the RM, hopefully spreading the containers around the cluster more evenly.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
