[
https://issues.apache.org/jira/browse/APEXCORE-10?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15109469#comment-15109469
]
Isha Arkatkar commented on APEXCORE-10:
---------------------------------------
Anti-affinity specified with strict policy will have precedence over relaxed
policy. In order to have precedence given for partitions anti-affinity, it
should be set to Strict. AM will try to honor all the anti-affinity
requirements specified with Strict policy.
Yes, partitioning will affect anti-affinity behavior. We need to handle
anti-affinity requirements in case dynamic partitioning in AM itself.
Rack locality is currently not supported in Apex. However, considering host
locality as example. For host-locality, we send container request to Yarn with
relaxLocality set to false i.e. host-locality request is always with Strict
policy. If anti-affinity policy is also Strict, AM will try to honor both
host-locality and anti-affinity.
In case of node crash, it is again Application Master's responsibility to
maintain anti-affinity requirements. AM should request container on other node
such that strict anti-affinity is still honored. In case there are no such
resources available, it would keep sending container requests to Yarn. This
behavior is similar to normal node crash scenario: AM requests Yarn for
containers allocated on crashed node. If enough resources are not available to
reallocate containers, AM will keep sending requests.
> Enable non-affinity of operators per node (not containers)
> ----------------------------------------------------------
>
> Key: APEXCORE-10
> URL: https://issues.apache.org/jira/browse/APEXCORE-10
> Project: Apache Apex Core
> Issue Type: Task
> Reporter: Amol Kekre
> Assignee: Isha Arkatkar
> Labels: roadmap
>
> The issue happens on cloud which provides virtual cores with software like
> Xen underneath. In effect if CPU intensive operators land up on same node we
> have a resource bottleneck,
> Need to create an attribute that does the following
> - Operators A & B should not be on same node
> - Stram should use this attribute to try to get containers on different node
> It is understood that the user is making an explicit choice to use NIC
> instead of stream local optimization
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
