[
https://issues.apache.org/jira/browse/APEXCORE-10?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15108221#comment-15108221
]
Milind Barve commented on APEXCORE-10:
--------------------------------------
A couple of questions -
Would anti-affinity for partitions take precedence over the anti-affinity of
operators? Can this be configured?
Wouldn't there be a cascading effect of partitioning on the anti-affinity of
operators?
In case of clashes between anti-affinity and locality, what takes precedence?
(For example, would the partitions be created across racks and different
operators remain in the nodes on the same rack, or would all the partitions be
within the same rack and the locality for operators will not be honored?)
What would happen if a node crashes and YARN allocates a node on which another
operator/partition is already active. YARN would not have any knowledge about
the anti-affinity. So what would be our approach in this case? Ideally, if we
get resources from YARN on a certain node, we shoul dbe able to run the
instance/operator on the same node. However, is there a guarantee that YARN
will allocate the exact resources we have asked for? Further even if it does,
would we honour the "strict" policy and prevent the instance/operator from
coming up or we bypass the same.
> Enable non-affinity of operators per node (not containers)
> ----------------------------------------------------------
>
> Key: APEXCORE-10
> URL: https://issues.apache.org/jira/browse/APEXCORE-10
> Project: Apache Apex Core
> Issue Type: Task
> Reporter: Amol Kekre
> Assignee: Isha Arkatkar
> Labels: roadmap
>
> The issue happens on cloud which provides virtual cores with software like
> Xen underneath. In effect if CPU intensive operators land up on same node we
> have a resource bottleneck,
> Need to create an attribute that does the following
> - Operators A & B should not be on same node
> - Stram should use this attribute to try to get containers on different node
> It is understood that the user is making an explicit choice to use NIC
> instead of stream local optimization
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
