Hi,
To support setting affinity rules from properties.xml, I added json
string codec to read the value. I had to wrap the list of rules in an
object call AffinityRuleSet to support easy serialization.
Here is an example for setting through property:
<property>
<name>dt.attr.AFFINITY_RULES_SET</name>
<value>
{
"affinityRules": [
{
"operatorRegex": "rand|console",
"locality": "NODE_LOCAL",
"type": "AFFINITY",
"relaxLocality": false
},
{
"operators": {
"first": "rand",
"second": "passThru"
},
"locality": "NODE_LOCAL",
"type": "ANTI_AFFINITY",
"relaxLocality": false
},
{
"operatorsList": [
"passThru",
"passThru"
],
"locality": "NODE_LOCAL",
"type": "ANTI_AFFINITY",
"relaxLocality": false
}
]
}
</value>
</property>
Thanks,
Isha
On Thu, Feb 25, 2016 at 8:16 PM, Chinmay Kolhatkar <[email protected]>
wrote:
> Hi Isha,
>
> 2 ways looks fine.
>
> Just one point, how would user set affinity rules using DAG content
> attribute from properties.xml file?
> Can you please provide some example for this?
>
> Thanks,
> Chinmay.
>
>
> On Fri, Feb 26, 2016 at 2:57 AM, Isha Arkatkar <[email protected]>
> wrote:
>
> > Thanks for the input Chinmay.
> >
> > The last parameter true/false is for relaxing the constraint. If true,
> the
> > affinity rule would be relaxed if constraint cannot be satisfied. And if
> > false, application will keep requesting and waiting to launch containers
> > till all rules are satisfied.
> >
> > Regarding the dag.setAffinity APIs. I can add those couple APIs in
> > LogicalPlan as a wrapper that internally can set the Dag Attributes.
> > Probably yes, that would be easier to use, to set affinity directly on
> dag
> > object.
> >
> > Then there would 2 ways to set affinity rules:
> >
> > 1. Through DAG context attribute (Either through code or through
> > properties.xml)
> > 2. By calling setAffinity on dag.
> >
> > Thanks,
> > Isha
> >
> > On Wed, Feb 24, 2016 at 10:11 PM, Chinmay Kolhatkar <[email protected]>
> > wrote:
> >
> > > I agree with Pramod. We should probably have single interface for
> > > specifying both if at all possible. This is in interest of not
> confusing
> > > users with two different configurations at the same time make a
> > > locality/affinity API to apear similar to what is out there in the word
> > at
> > > the moment.
> > >
> > > Though it might be early to deprecate stream locality, but after some
> > time
> > > based on user's experience we should consider deprecating Stream
> Locality
> > > API.
> > >
> > > @Isha, nice APIs. Questions/suggestions regarding this:
> > > 1. What does the "false" signify for AffinityRule constructor.
> > > 2. This is a suggestion. Instead of having a List<AffinityRule> and
> then
> > > setting this object to attribute of the DAG, should we have an API in
> > > LogicalPlan like following:
> > >
> > > dag.setAffinity(<operator1>, <operator2>, <locality_type>, false/true)
> > >
> > > dag.setAntiAffinity(<operator1>, <operator2>, false/true) // Not sure
> > > what false/true mean, added for completion. But the point here is to
> > remove
> > > locality for anti-affinity.
> > >
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > >
> > > On Thu, Feb 25, 2016 at 5:17 AM, Pramod Immaneni <
> [email protected]
> > >
> > > wrote:
> > >
> > > > You could specify it by specifying the names of all operators that
> are
> > > part
> > > > of the stream.
> > > >
> > > > On Wed, Feb 24, 2016 at 3:44 PM, Amol Kekre <[email protected]>
> > > wrote:
> > > >
> > > > > These are two diff ways of specifying something similar, but not
> > always
> > > > > same. For example, how do I specify the following
> > > > >
> > > > > This stream has way to high throughput. Force all operators that
> > > connect
> > > > to
> > > > > it today (or in future) to be container local.
> > > > >
> > > > > With stream locality I get that as operator ports are added to the
> > > > stream,
> > > > > they automatically inherit stream locality.
> > > > >
> > > > > Thks,
> > > > > Amol
> > > > >
> > > > > On Wed, Feb 24, 2016 at 3:37 PM, Pramod Immaneni <
> > > [email protected]
> > > > >
> > > > > wrote:
> > > > >
> > > > > > To explain the affinity support below seems to be a super-set
> that
> > > can
> > > > > > consume the stream locality setting whether that was the original
> > > > > intention
> > > > > > or not.
> > > > > >
> > > > > > On Wed, Feb 24, 2016 at 3:35 PM, Pramod Immaneni <
> > > > [email protected]
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Why would we want two different ways of specifying the same
> > thing?
> > > > > > >
> > > > > > > On Wed, Feb 24, 2016 at 3:29 PM, Amol Kekre <
> > [email protected]>
> > > > > > wrote:
> > > > > > >
> > > > > > >> Stream locality is a stream attribute. Affinity is operator
> > > > attribute.
> > > > > > >> Technically they may not be connected. Though operator
> affinity
> > > can
> > > > be
> > > > > > >> extended to specify a situation that implicity covers stream
> > > > locality,
> > > > > > it
> > > > > > >> may be better to let user say the two following statements
> > > > > > independently.
> > > > > > >>
> > > > > > >> - This stream is heavy on I/O and I want it to be ..._local //
> > > This
> > > > > > >> statement is independent of future additions of operators to
> > this
> > > > > stream
> > > > > > >> - I need these logical operators to not be (or be) on the same
> > > node
> > > > > > >>
> > > > > > >> Do note that container_local or node_local stream localities
> > only
> > > > > relate
> > > > > > >> to
> > > > > > >> connected operators.
> > > > > > >>
> > > > > > >> Thks
> > > > > > >> Amol
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Feb 24, 2016 at 1:54 PM, Pramod Immaneni <
> > > > > > [email protected]>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Would it make sense to deprecate stream locality in favor of
> > > this?
> > > > > > >> >
> > > > > > >> > On Tue, Feb 23, 2016 at 6:05 PM, Isha Arkatkar <
> > > > > [email protected]>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hi Pramod,
> > > > > > >> > >
> > > > > > >> > > We can have a list or operators or regex to specify
> > > operators
> > > > > > list
> > > > > > >> for
> > > > > > >> > > affinity rule. Though, internal implementation will
> > translate
> > > > both
> > > > > > >> these
> > > > > > >> > > into list of pairs. Since, having pair of operators helps
> in
> > > > > > >> validation
> > > > > > >> > > phase.
> > > > > > >> > >
> > > > > > >> > > And yes, validation phase will catch conflicting locality
> > or
> > > > > > affinity
> > > > > > >> > > rules and throw validation exception.
> > > > > > >> > >
> > > > > > >> > > Thanks for input regarding unifiers. It is probably
> alright
> > > to
> > > > > not
> > > > > > >> > include
> > > > > > >> > > it in the affinity settings. Will leave the current
> > > > implementation
> > > > > > as
> > > > > > >> is.
> > > > > > >> > >
> > > > > > >> > > Thanks!
> > > > > > >> > > Isha
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Tue, Feb 23, 2016 at 5:56 PM, Pramod Immaneni <
> > > > > > >> [email protected]
> > > > > > >> > >
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Isha,
> > > > > > >> > > >
> > > > > > >> > > > If the implementation would support (eventually)
> affinity
> > > > > between
> > > > > > >> more
> > > > > > >> > > than
> > > > > > >> > > > 2 logical operators, instead of relying on specifying it
> > as
> > > > > > multiple
> > > > > > >> > > rules
> > > > > > >> > > > why not make provision in the API to specify more than 2
> > > > logical
> > > > > > >> > > operators?
> > > > > > >> > > > If there is a stream locality rule that conflicts with
> the
> > > > > > affinity
> > > > > > >> > rule
> > > > > > >> > > > will the validation catch this? With unifiers the common
> > > case
> > > > of
> > > > > > MxN
> > > > > > >> > > > locality is already chosen based on downstream operator.
> > In
> > > > > other
> > > > > > >> cases
> > > > > > >> > > > such as single downstream or cascade case I am not sure
> it
> > > is
> > > > > all
> > > > > > >> the
> > > > > > >> > > > important to have it.
> > > > > > >> > > >
> > > > > > >> > > > Thanks
> > > > > > >> > > >
> > > > > > >> > > > On Tue, Feb 23, 2016 at 5:37 PM, Isha Arkatkar <
> > > > > > >> [email protected]>
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi all,
> > > > > > >> > > > >
> > > > > > >> > > > > I have opened a review only pull request for
> > handling
> > > > > > affinity
> > > > > > >> > > rules
> > > > > > >> > > > in
> > > > > > >> > > > > Apex:
> > > > > > >> > > > >
> > https://github.com/apache/incubator-apex-core/pull/234
> > > > > > >> > > > >
> > > > > > >> > > > > Just wanted to confirm with everyone, if the APIs for
> > > > > > specifying
> > > > > > >> > > > affinity
> > > > > > >> > > > > rules look alright. Also, wanted to get views on how
> > > > unifiers
> > > > > > >> should
> > > > > > >> > be
> > > > > > >> > > > > handled in case of operator affinity rules. More
> details
> > > > > follow:
> > > > > > >> > > > >
> > > > > > >> > > > > I have added a list of affinity rules as an attribute
> > in
> > > > Dag
> > > > > > >> > context.
> > > > > > >> > > > For
> > > > > > >> > > > > now, have added AffinityRule to be specified for
> > operator
> > > > > pairs.
> > > > > > >> We
> > > > > > >> > can
> > > > > > >> > > > add
> > > > > > >> > > > > regex support in next iteration.
> > > > > > >> > > > >
> > > > > > >> > > > > Here is sample usage:
> > > > > > >> > > > >
> > > > > > >> > > > > List<AffinityRule> rules = new ArrayList<>();
> > > > > > >> > > > > // To add node locality between 2 not connected
> > > > operators
> > > > > > rand
> > > > > > >> > and
> > > > > > >> > > > > console:
> > > > > > >> > > > > rules.add(new AffinityRule(Type.AFFINITY, new
> > > > OperatorPair(
> > > > > > >> > "rand",
> > > > > > >> > > > > "console"), Locality.NODE_LOCAL, false));
> > > > > > >> > > > >
> > > > > > >> > > > > // To add anti-affinity between all partitions of
> > rand
> > > > > > operator
> > > > > > >> > with
> > > > > > >> > > > all
> > > > > > >> > > > > partitions of passThru operator:
> > > > > > >> > > > > rules.add(new AffinityRule(Type.ANTI_AFFINITY, new
> > > > > > >> OperatorPair(
> > > > > > >> > > > "rand",
> > > > > > >> > > > > "passThru"), Locality.NODE_LOCAL, false));
> > > > > > >> > > > >
> > > > > > >> > > > > // To add anti-affinity between partitions of
> > passThru
> > > > > > >> operator,
> > > > > > >> > > give
> > > > > > >> > > > > same operator name in pair:
> > > > > > >> > > > > rules.add(new AffinityRule(Type.ANTI_AFFINITY, new
> > > > > > >> OperatorPair(
> > > > > > >> > > > > "passThru", "passThru"), Locality.NODE_LOCAL, false));
> > > > > > >> > > > >
> > > > > > >> > > > > // Set the rules in dag context
> > > > > > >> > > > > dag.setAttribute(DAGContext.AFFINITY_RULES, rules);
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > Please find a sample application using these affinity
> > > rules
> > > > > > here:
> > > > > > >> > > > >
> > > > > > >>
> > > https://github.com/ishark/Apex-Samples/tree/master/affinity-example
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > The actual implementation of affinity rules heavily
> > > depends
> > > > > node
> > > > > > >> > > specific
> > > > > > >> > > > > requests, that was already being handled in
> > > > > > >> > StreamingAppMasterService.
> > > > > > >> > > To
> > > > > > >> > > > > handle node requests for cloudera, I have added an
> > > override
> > > > to
> > > > > > >> > > blacklist
> > > > > > >> > > > > all other nodes except the ones where container
> request
> > is
> > > > to
> > > > > be
> > > > > > >> > > issued.
> > > > > > >> > > > >
> > > > > > >> > > > > There is one open question I wanted to bring up: In
> case
> > > of
> > > > > > >> > partitioned
> > > > > > >> > > > > operators, should the unifiers also follow affinity or
> > > > > > >> anti-affinity
> > > > > > >> > > > rules?
> > > > > > >> > > > > Or should they be independent. For now, I have kept
> them
> > > > > > >> independent
> > > > > > >> > > and
> > > > > > >> > > > > only the actual operators follow affinity rules.
> Please
> > > > > suggest
> > > > > > >> what
> > > > > > >> > > > would
> > > > > > >> > > > > make more sense from user's specification perspective.
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > Isha
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > On Mon, Feb 1, 2016 at 2:12 PM, Isha Arkatkar <
> > > > > > >> [email protected]>
> > > > > > >> > > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Hi folks,
> > > > > > >> > > > > >
> > > > > > >> > > > > > Summarizing the proposal for
> affinity/anti-affinity
> > > > rules
> > > > > > in
> > > > > > >> > apex
> > > > > > >> > > as
> > > > > > >> > > > > > per discussions in this mail thread. Please suggest
> > if I
> > > > > > missed
> > > > > > >> > > > > something.
> > > > > > >> > > > > >
> > > > > > >> > > > > > * For configuration:*
> > > > > > >> > > > > >
> > > > > > >> > > > > > - We will have application level
> > > affinity/anti-affinity
> > > > > > rules.
> > > > > > >> > For
> > > > > > >> > > > the
> > > > > > >> > > > > > first iteration, we can support for (anti-)affinity
> > > among
> > > > > > >> operators
> > > > > > >> > > > > within
> > > > > > >> > > > > > single application. We can revisit it in the next
> > > > iteration
> > > > > to
> > > > > > >> > > support
> > > > > > >> > > > > > across applications.
> > > > > > >> > > > > >
> > > > > > >> > > > > > - Each rule will consist of 4 components:
> > > > > > >> > > > > > <List of operators>, AFFINITY/ANTI-AFFINITY,
> > > > > > >> STRICT/RELAXED
> > > > > > >> > > > > policy,
> > > > > > >> > > > > > CONTAINER/NODE/RACK
> > > > > > >> > > > > > I have checked that Apex supports container locality
> > > > > between 2
> > > > > > >> > > > operators
> > > > > > >> > > > > > that are not connected by Steam. Though did not find
> > an
> > > > API
> > > > > > for
> > > > > > >> > > setting
> > > > > > >> > > > > it
> > > > > > >> > > > > > through application. So, affinity rules can cover
> > > > container
> > > > > > >> > affinity
> > > > > > >> > > as
> > > > > > >> > > > > > well.
> > > > > > >> > > > > > We will not support THREAD locality in app level
> > > affinity
> > > > or
> > > > > > >> > > > > > anti-affinity, since it does not apply to
> unconnected
> > > > > > operators.
> > > > > > >> > > > > >
> > > > > > >> > > > > > The list of operators can also be specified by
> regex
> > > > > > >> > > > > >
> > > > > > >> > > > > > *For Implementation: *
> > > > > > >> > > > > >
> > > > > > >> > > > > > - I have verified that Node specific requests are
> > > > honored
> > > > > by
> > > > > > >> > > > > HortonWorks
> > > > > > >> > > > > > for both Fair and Capacity scheduler. So we can go
> > with
> > > > the
> > > > > > >> > approach
> > > > > > >> > > > of
> > > > > > >> > > > > > node specific container requests to implement
> > affinity
> > > > > rules.
> > > > > > >> > > > > >
> > > > > > >> > > > > > - However, we may need to fall back on blacklist
> > > approach
> > > > > for
> > > > > > >> CDH
> > > > > > >> > > > > distro,
> > > > > > >> > > > > > as node specific requests may not work well. As:
> > > > > > >> > > > > > -> With Fair scheduler, node specific
> > requests
> > > > are
> > > > > > not
> > > > > > >> > > honored
> > > > > > >> > > > > in
> > > > > > >> > > > > > CDH 4. Application master keeps sending requests to
> > > Yarn,
> > > > > but
> > > > > > no
> > > > > > >> > > > > containers
> > > > > > >> > > > > > are allocated.
> > > > > > >> > > > > > -> I could not yet verify the same with
> Fair
> > > and
> > > > > > >> Capacity
> > > > > > >> > > > > > Scheduler on CDH 5. Ran into some configuration
> issues
> > > > when
> > > > > > >> setting
> > > > > > >> > > up
> > > > > > >> > > > > CDH
> > > > > > >> > > > > > 5.5.1. Will update the thread with findings.
> > > > > > >> > > > > >
> > > > > > >> > > > > > I will try to build a prototype for affinity rules
> > > with
> > > > > > these
> > > > > > >> > > things
> > > > > > >> > > > in
> > > > > > >> > > > > > mind. Please let me know if anyone would like to
> > > > collaborate
> > > > > > on
> > > > > > >> > this
> > > > > > >> > > > > > effort! :)
> > > > > > >> > > > > >
> > > > > > >> > > > > > Thanks!
> > > > > > >> > > > > > Isha
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Mon, Jan 25, 2016 at 10:23 AM, Vlad Rozov <
> > > > > > >> > > [email protected]>
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > >> Yes, with ability to refer not only to a host name,
> > but
> > > > to
> > > > > > >> > operators
> > > > > > >> > > > > >> (possibly both in the same and other applications).
> > In
> > > > > > general
> > > > > > >> > this
> > > > > > >> > > > > should
> > > > > > >> > > > > >> allow to specify both affinity and anti affinity
> > using
> > > > > > >> > expressions.
> > > > > > >> > > > For
> > > > > > >> > > > > >> example LOCALITY_CONTAINER = "{app1.op1 or
> app1.op2}
> > > and
> > > > > > >> > > {!app2.op1}".
> > > > > > >> > > > > >>
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> On 1/25/16 09:39, Thomas Weise wrote:
> > > > > > >> > > > > >>
> > > > > > >> > > > > >>> Agreed. Originally I thought you wanted both
> > operators
> > > > on
> > > > > > the
> > > > > > >> > same
> > > > > > >> > > > > host.
> > > > > > >> > > > > >>> To
> > > > > > >> > > > > >>> pin them to the same container the same idea can
> be
> > > > > applied,
> > > > > > >> > > through
> > > > > > >> > > > a
> > > > > > >> > > > > >>> "LOCALITY_CONTAINER" attribute.
> > > > > > >> > > > > >>>
> > > > > > >> > > > > >>>
> > > > > > >> > > > > >>> On Mon, Jan 25, 2016 at 9:18 AM, Vlad Rozov <
> > > > > > >> > > [email protected]
> > > > > > >> > > > >
> > > > > > >> > > > > >>> wrote:
> > > > > > >> > > > > >>>
> > > > > > >> > > > > >>> Works as a workaround and requires two extra ports
> > > that
> > > > > > >> operators
> > > > > > >> > > > > >>>> designer
> > > > > > >> > > > > >>>> may not necessarily provide out of the box. We
> may
> > > > apply
> > > > > > the
> > > > > > >> > same
> > > > > > >> > > > hack
> > > > > > >> > > > > >>>> to
> > > > > > >> > > > > >>>> anti-affinity and request that any two operators
> > with
> > > > > > >> > > anti-affinity
> > > > > > >> > > > > rule
> > > > > > >> > > > > >>>> are connected by non functional stream, but it
> does
> > > not
> > > > > > look
> > > > > > >> > like
> > > > > > >> > > a
> > > > > > >> > > > > good
> > > > > > >> > > > > >>>> option :).
> > > > > > >> > > > > >>>>
> > > > > > >> > > > > >>>>
> > > > > > >> > > > > >>>>
> > > > > > >> > > > > >>>> On 1/25/16 08:52, Thomas Weise wrote:
> > > > > > >> > > > > >>>>
> > > > > > >> > > > > >>>> Sandeep suggested to connect them with a stream.
> > > > > > >> > > > > >>>>>
> > > > > > >> > > > > >>>>> On Mon, Jan 25, 2016 at 8:49 AM, Vlad Rozov <
> > > > > > >> > > > [email protected]
> > > > > > >> > > > > >
> > > > > > >> > > > > >>>>> wrote:
> > > > > > >> > > > > >>>>>
> > > > > > >> > > > > >>>>> Sorry, possibly I miss something. X and Z are
> not
> > > > > directly
> > > > > > >> > > > connected
> > > > > > >> > > > > >>>>> by a
> > > > > > >> > > > > >>>>>
> > > > > > >> > > > > >>>>>> stream, how can I apply THREAD_LOCAL?
> > > > > > >> > > > > >>>>>>
> > > > > > >> > > > > >>>>>>
> > > > > > >> > > > > >>>>>> On 1/25/16 08:44, Thomas Weise wrote:
> > > > > > >> > > > > >>>>>>
> > > > > > >> > > > > >>>>>> Why does THREAD_LOCAL not work?
> > > > > > >> > > > > >>>>>>
> > > > > > >> > > > > >>>>>>> On Mon, Jan 25, 2016 at 7:43 AM, Vlad Rozov <
> > > > > > >> > > > > [email protected]
> > > > > > >> > > > > >>>>>>> >
> > > > > > >> > > > > >>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>
> > > > > > >> > > > > >>>>>>> Neither LOCALITY_HOST or THREAD_LOCAL provides
> > > > > required
> > > > > > >> > > > > >>>>>>> functionality.
> > > > > > >> > > > > >>>>>>> The
> > > > > > >> > > > > >>>>>>>
> > > > > > >> > > > > >>>>>>> goal is to deploy X and Z to the same
> > > > container/JVM. X
> > > > > > >> may be
> > > > > > >> > > > > >>>>>>>> deployed
> > > > > > >> > > > > >>>>>>>> on
> > > > > > >> > > > > >>>>>>>> any node. Z needs to be deployed on the same
> > > > > container
> > > > > > >> as X,
> > > > > > >> > > so
> > > > > > >> > > > X
> > > > > > >> > > > > >>>>>>>> does
> > > > > > >> > > > > >>>>>>>> not
> > > > > > >> > > > > >>>>>>>> have any host affinity, and Z has affinity to
> > X,
> > > > not
> > > > > to
> > > > > > >> any
> > > > > > >> > > > > specific
> > > > > > >> > > > > >>>>>>>> host.
> > > > > > >> > > > > >>>>>>>> If database client caches writes, reading
> > inside
> > > > the
> > > > > > same
> > > > > > >> > JVM
> > > > > > >> > > > will
> > > > > > >> > > > > >>>>>>>> be
> > > > > > >> > > > > >>>>>>>> in
> > > > > > >> > > > > >>>>>>>> process lookup, while deploying Z to any
> other
> > > > > host/JVM
> > > > > > >> > leads
> > > > > > >> > > to
> > > > > > >> > > > > >>>>>>>> inter
> > > > > > >> > > > > >>>>>>>> process or inter host lookup.
> > > > > > >> > > > > >>>>>>>>
> > > > > > >> > > > > >>>>>>>>
> > > > > > >> > > > > >>>>>>>> On 1/24/16 21:03, Thomas Weise wrote:
> > > > > > >> > > > > >>>>>>>>
> > > > > > >> > > > > >>>>>>>> There are the attributes LOCALITY_HOST and
> > > > > > LOCALITY_RACK
> > > > > > >> for
> > > > > > >> > > an
> > > > > > >> > > > > >>>>>>>> operator
> > > > > > >> > > > > >>>>>>>>
> > > > > > >> > > > > >>>>>>>> to
> > > > > > >> > > > > >>>>>>>>> achieve precisely what you are describing.
> The
> > > > value
> > > > > > >> would
> > > > > > >> > be
> > > > > > >> > > > an
> > > > > > >> > > > > >>>>>>>>> alias
> > > > > > >> > > > > >>>>>>>>> that
> > > > > > >> > > > > >>>>>>>>> can be mapped to physical hosts for
> > allocation.
> > > > The
> > > > > > same
> > > > > > >> > > could
> > > > > > >> > > > be
> > > > > > >> > > > > >>>>>>>>> used
> > > > > > >> > > > > >>>>>>>>> of
> > > > > > >> > > > > >>>>>>>>> anti-affinity, by assigning different
> values.
> > > > > > >> > > > > >>>>>>>>>
> > > > > > >> > > > > >>>>>>>>> Stream locality is a special case of
> > expressing
> > > > > > affinity
> > > > > > >> > for
> > > > > > >> > > 2
> > > > > > >> > > > > >>>>>>>>> operators.
> > > > > > >> > > > > >>>>>>>>> Note that we also need a way to extend this
> to
> > > > > > >> partitions
> > > > > > >> > of
> > > > > > >> > > > the
> > > > > > >> > > > > >>>>>>>>> same
> > > > > > >> > > > > >>>>>>>>> operator.
> > > > > > >> > > > > >>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>
> > > > > > >> > > > > >>>>>>>>> On Sun, Jan 24, 2016 at 8:52 PM, Vlad Rozov
> <
> > > > > > >> > > > > >>>>>>>>> [email protected]>
> > > > > > >> > > > > >>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>
> > > > > > >> > > > > >>>>>>>>> It may be quite extreme use case, but
> suppose
> > > that
> > > > > > >> > operator X
> > > > > > >> > > > > >>>>>>>>> writes
> > > > > > >> > > > > >>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>> a
> > > > > > >> > > > > >>>>>>>>>
> > > > > > >> > > > > >>>>>>>>> distributed in-memory database and operator
> Z
> > > > reads
> > > > > > from
> > > > > > >> > it.
> > > > > > >> > > Z
> > > > > > >> > > > is
> > > > > > >> > > > > >>>>>>>>> not
> > > > > > >> > > > > >>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>> directly connected to X. In such cases it
> may
> > > be
> > > > > > >> necessary
> > > > > > >> > > to
> > > > > > >> > > > > let
> > > > > > >> > > > > >>>>>>>>>> an
> > > > > > >> > > > > >>>>>>>>>> application request container affinity
> > > (deploying
> > > > > to
> > > > > > >> the
> > > > > > >> > > same
> > > > > > >> > > > > JVM)
> > > > > > >> > > > > >>>>>>>>>> for
> > > > > > >> > > > > >>>>>>>>>> operators X and Z as writes may be cached
> and
> > > > reads
> > > > > > >> from
> > > > > > >> > the
> > > > > > >> > > > > same
> > > > > > >> > > > > >>>>>>>>>> JVM
> > > > > > >> > > > > >>>>>>>>>> will
> > > > > > >> > > > > >>>>>>>>>> be potentially faster.
> > > > > > >> > > > > >>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>> Technically the same may be applied to
> thread
> > > > local
> > > > > > on
> > > > > > >> > NUMA
> > > > > > >> > > > > >>>>>>>>>> boxes, as
> > > > > > >> > > > > >>>>>>>>>> NUMA
> > > > > > >> > > > > >>>>>>>>>> aware deployment of containers is not
> > > supported.
> > > > > > >> > > > > >>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>> Vlad
> > > > > > >> > > > > >>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>> On 1/23/16 18:01, Yogi Devendra wrote:
> > > > > > >> > > > > >>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>> @Isha
> > > > > > >> > > > > >>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>> In my opinion:
> > > > > > >> > > > > >>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>> THREAD_LOCAL, CONTAINER_LOCAL on stream
> is a
> > > > > special
> > > > > > >> case
> > > > > > >> > > of
> > > > > > >> > > > > >>>>>>>>>>> generic
> > > > > > >> > > > > >>>>>>>>>>> rules
> > > > > > >> > > > > >>>>>>>>>>> for Operator X and Operator Y.
> > > > > > >> > > > > >>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>> We can say that, THREAD_LOCAL,
> > CONTAINER_LOCAL
> > > > > would
> > > > > > >> be
> > > > > > >> > > > > >>>>>>>>>>> applicable
> > > > > > >> > > > > >>>>>>>>>>> only
> > > > > > >> > > > > >>>>>>>>>>> if
> > > > > > >> > > > > >>>>>>>>>>> operator X and Y are connected by stream.
> > But,
> > > > way
> > > > > > to
> > > > > > >> > > express
> > > > > > >> > > > > >>>>>>>>>>> this
> > > > > > >> > > > > >>>>>>>>>>> should
> > > > > > >> > > > > >>>>>>>>>>> be similar to other rules for affinity.
> > > > > > >> > > > > >>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>> ~ Yogi
> > > > > > >> > > > > >>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>> On 24 January 2016 at 03:49, Isha
> Arkatkar <
> > > > > > >> > > > > [email protected]
> > > > > > >> > > > > >>>>>>>>>>> >
> > > > > > >> > > > > >>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>> Hey Chinmay,
> > > > > > >> > > > > >>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>> I certainly agree on common set of
> > > rules
> > > > > for
> > > > > > >> > > > > configuring
> > > > > > >> > > > > >>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>> affinity!
> > > > > > >> > > > > >>>>>>>>>>>> Well
> > > > > > >> > > > > >>>>>>>>>>>> put by a concrete example. :)
> > > > > > >> > > > > >>>>>>>>>>>> Only thing I would like to point
> > is:
> > > > > > >> affinity of
> > > > > > >> > > > > >>>>>>>>>>>> operators
> > > > > > >> > > > > >>>>>>>>>>>> should
> > > > > > >> > > > > >>>>>>>>>>>> not
> > > > > > >> > > > > >>>>>>>>>>>> cover thread and container locality.
> Since
> > > this
> > > > > is
> > > > > > >> only
> > > > > > >> > > > > >>>>>>>>>>>> applicable
> > > > > > >> > > > > >>>>>>>>>>>> when
> > > > > > >> > > > > >>>>>>>>>>>> operators are connected by stream. So, it
> > > makes
> > > > > > >> sense to
> > > > > > >> > > > have
> > > > > > >> > > > > >>>>>>>>>>>> it on
> > > > > > >> > > > > >>>>>>>>>>>> Stream
> > > > > > >> > > > > >>>>>>>>>>>> rather than in common configuration.
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> And yes, DAG.validate should only
> > > check
> > > > > for
> > > > > > >> > > REQUIRED
> > > > > > >> > > > or
> > > > > > >> > > > > >>>>>>>>>>>> STRICT
> > > > > > >> > > > > >>>>>>>>>>>> policy. We
> > > > > > >> > > > > >>>>>>>>>>>> can agree on one of the terminologies
> > > > > > STRICT/RELAXED
> > > > > > >> or
> > > > > > >> > > > > >>>>>>>>>>>> REQUIRED/PREFERRED.
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> Thanks!
> > > > > > >> > > > > >>>>>>>>>>>> Isha
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> On Fri, Jan 22, 2016 at 8:38 PM, Chinmay
> > > > > Kolhatkar
> > > > > > <
> > > > > > >> > > > > >>>>>>>>>>>> [email protected]>
> > > > > > >> > > > > >>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> Hi Isha, Bhupesh,
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> When I suggested singe affinity rule, I
> was
> > > > > mainly
> > > > > > >> > talking
> > > > > > >> > > > > about
> > > > > > >> > > > > >>>>>>>>>>>> "how
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> to"
> > > > > > >> > > > > >>>>>>>>>>>>> of configuration and not of
> > implementation.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> I see locality is in a way suggesting an
> > > > > affinity
> > > > > > of
> > > > > > >> > > > > operators.
> > > > > > >> > > > > >>>>>>>>>>>>> They're
> > > > > > >> > > > > >>>>>>>>>>>>> close terminologies.
> > > > > > >> > > > > >>>>>>>>>>>>> By configuring a locality on stream,
> we're
> > > > also,
> > > > > > in
> > > > > > >> a
> > > > > > >> > > way,
> > > > > > >> > > > > >>>>>>>>>>>>> defining
> > > > > > >> > > > > >>>>>>>>>>>>> affinity of operators.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> Until now, only locality was there and
> > hence
> > > > was
> > > > > > >> > straight
> > > > > > >> > > > > >>>>>>>>>>>>> forward
> > > > > > >> > > > > >>>>>>>>>>>>> in
> > > > > > >> > > > > >>>>>>>>>>>>> configuration for user.
> > > > > > >> > > > > >>>>>>>>>>>>> Tomorrow, when anti-affinity
> configuration
> > > > comes
> > > > > > up,
> > > > > > >> > one
> > > > > > >> > > > > might
> > > > > > >> > > > > >>>>>>>>>>>>> get
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> confused
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> on how to best use both locality &
> > > > > anti-affinity.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> Hence suggested to make both
> > > > (locality/affinity &
> > > > > > >> > > > > >>>>>>>>>>>> anti-affinity) as
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> a
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> part
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> of single configuration.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> Suggestion is to have a more commonly
> > adopted
> > > > > > >> > > configuration
> > > > > > >> > > > > >>>>>>>>>>>> which
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> admins
> > > > > > >> > > > > >>>>>>>>>>>>> and developer's are familiar with.
> > > > > > >> > > > > >>>>>>>>>>>>> Again referring to vShere Hypervisor's
> > > > affinity
> > > > > > >> rules.
> > > > > > >> > I
> > > > > > >> > > > > think
> > > > > > >> > > > > >>>>>>>>>>>>> they
> > > > > > >> > > > > >>>>>>>>>>>>> have
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> a
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> single configuration which does both.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> Having said that, here is a quick
> > suggestion
> > > on
> > > > > how
> > > > > > >> both
> > > > > > >> > > can
> > > > > > >> > > > > be
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> achieved
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> in
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> a single configuration:
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> CATEGORY TYPE POLICY
> > > > > > >> ENTITIES
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> Affinity THREAD
> REQUIRED
> > > > > > >> O1, O2
> > > > > > >> > > > > >>>>>>>>>>>>> //Meaning
> > > > > > >> > > > > >>>>>>>>>>>>> Operator1 & Operator2 should be thread
> > local
> > > > > > >> > > > > >>>>>>>>>>>>> Affinity NODE
> > PREFERRED
> > > > > > >> O3, O4
> > > > > > >> > > > > >>>>>>>>>>>>> //Meaning
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> O3 &
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> O4 are preferred to be in node local
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> AntiAffinity NODE REQUIRED
> > > > > > O1,
> > > > > > >> O4
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> //Meaning
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> O1 &
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> O4 should not be on same node.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> AntiAffinity RACK
> PREFERRED
> > > > > O2,
> > > > > > O4
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> //Meaning
> > > > > > >> > > > > >>>>>>>>>>>>> O2
> > > > > > >> > > > > >>>>>>>>>>>>> &
> > > > > > >> > > > > >>>>>>>>>>>>> O4 are preferred not to be on same rack.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> Linux setting affinity of CPUs for
> threads
> > > is
> > > > > > >> another
> > > > > > >> > way
> > > > > > >> > > > of
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> configuration
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> we can take a look at.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> Learning from these commonly adopted
> > > > > configuration
> > > > > > >> > > pattern,
> > > > > > >> > > > we
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> should
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> come
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> up with best configuration suitable for
> > > > > > distributed
> > > > > > >> > > > > >>>>>>>>>>>>> environment.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> Idea here is to not have our own
> > > configuration
> > > > > and
> > > > > > >> give
> > > > > > >> > > > > >>>>>>>>>>>> something
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> new
> > > > > > >> > > > > >>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>> the users. Otherwise such an important
> > > concept
> > > > > > might
> > > > > > >> > > > quickly
> > > > > > >> > > > > >>>>>>>>>>>>> get
> > > > > > >> > > > > >>>>>>>>>>>>> lost.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> Regarding the DAG.validate I think we
> > would
> > > > need
> > > > > > to
> > > > > > >> add
> > > > > > >> > > > some
> > > > > > >> > > > > >>>>>>>>>>>>> new
> > > > > > >> > > > > >>>>>>>>>>>>> stuff
> > > > > > >> > > > > >>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>> take care of anti-affinity.
> > > > > > >> > > > > >>>>>>>>>>>>> Plus, anti-affinity/affinity should be
> > > > validated
> > > > > > at
> > > > > > >> > > > > >>>>>>>>>>>>> DAG.validate
> > > > > > >> > > > > >>>>>>>>>>>>> only
> > > > > > >> > > > > >>>>>>>>>>>>> for
> > > > > > >> > > > > >>>>>>>>>>>>> the ones which are required.
> > > > > > >> > > > > >>>>>>>>>>>>> For preferred policies, validation in
> > > logical
> > > > > plan
> > > > > > >> > might
> > > > > > >> > > > be a
> > > > > > >> > > > > >>>>>>>>>>>>> early
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> check.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> Thanks,
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>> Chinmay.
> > > > > > >> > > > > >>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> On Sat, Jan 23, 2016 at 3:15 AM, Isha
> > > > Arkatkar <
> > > > > > >> > > > > >>>>>>>>>>>>> [email protected]>
> > > > > > >> > > > > >>>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> Hi,
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> Thanks for inputs! I like the
> idea
> > of
> > > > > > having
> > > > > > >> > > single
> > > > > > >> > > > > set
> > > > > > >> > > > > >>>>>>>>>>>>> of
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> (anti?)affinity rules at dag context
> > level.
> > > > > > >> > > > > >>>>>>>>>>>>>> There could still be conflicts
> > based
> > > > on
> > > > > > >> > Locality
> > > > > > >> > > > for
> > > > > > >> > > > > >>>>>>>>>>>>>> Streams
> > > > > > >> > > > > >>>>>>>>>>>>>> or
> > > > > > >> > > > > >>>>>>>>>>>>>> Node
> > > > > > >> > > > > >>>>>>>>>>>>>> locality attribute set for operators.
> But
> > > as
> > > > > > >> Sandeep
> > > > > > >> > > > > >>>>>>>>>>>>>> suggested, I
> > > > > > >> > > > > >>>>>>>>>>>>>> also
> > > > > > >> > > > > >>>>>>>>>>>>>> think Dag.validate should fail in case
> of
> > > > > > >> > contradicting
> > > > > > >> > > > > >>>>>>>>>>>>>> constraints.
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> We currently do not have
> 'affinity'
> > > > among
> > > > > > non
> > > > > > >> > > stream
> > > > > > >> > > > > >>>>>>>>>>>>>> operators, as
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> Bhupesh
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> pointed out. It is somewhat achievable
> by
> > > > > > >> requesting
> > > > > > >> > > node
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> locality
> > > > > > >> > > > > >>>>>>>>>>>>> for
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> operators (Assuming node requests work
> as
> > > > > > expected).
> > > > > > >> > But
> > > > > > >> > > > > >>>>>>>>>>>>> should we
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> consider
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> adding affinity specifications support
> as
> > > > well
> > > > > > >> along
> > > > > > >> > > with
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> anti-affinity?
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> Regarding specification of
> > attributes
> > > > > from
> > > > > > >> > > > > dt-site.xml.
> > > > > > >> > > > > >>>>>>>>>>>>>> We
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> can
> > > > > > >> > > > > >>>>>>>>>>>>> go
> > > > > > >> > > > > >>>>>>>>>>>>> with
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> Json like string or even xml
> > representation
> > > > for
> > > > > > >> complex
> > > > > > >> > > > > >>>>>>>>>>>>> objects.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> What
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> is
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> our current behavior for setting Java
> > > object
> > > > > > >> > properties
> > > > > > >> > > > > >>>>>>>>>>>>>> through
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> XML?
> > > > > > >> > > > > >>>>>>>>>>>>> We
> > > > > > >> > > > > >>>>>>>>>>>>> can
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> follow the same for this as well.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> As for precedence or ability to
> > > satisfy
> > > > > > >> > > constraints:
> > > > > > >> > > > > >>>>>>>>>>>>> Right
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> now in
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> normal
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> scenario, if resources are not
> available
> > > for
> > > > > > >> > allocating
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> containers,
> > > > > > >> > > > > >>>>>>>>>>>>> we
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> keep
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> sending to request till all are
> obtained.
> > > > > > >> Likewise, in
> > > > > > >> > > > case
> > > > > > >> > > > > of
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> strict
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> anti-affinity policy, we should keep the
> > > > > > >> application in
> > > > > > >> > > > > >>>>>>>>>>>>> ACCEPTED
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> state
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> till
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> the anti-affinity constraint is
> > satisfied.
> > > > For
> > > > > > >> relaxed
> > > > > > >> > > > > >>>>>>>>>>>>>> policy, we
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> can
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> decide timeout for relaxing the
> > > anti-affinity
> > > > > > rule.
> > > > > > >> > > Please
> > > > > > >> > > > > note
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> this
> > > > > > >> > > > > >>>>>>>>>>>>>> applies only when we have
> > non-contradicting
> > > > > > rules.
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> Thanks!
> > > > > > >> > > > > >>>>>>>>>>>>>> Isha
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 5:30 AM,
> Bhupesh
> > > > > Chawda <
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> [email protected]
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> I agree on having a single set of rules
> > for
> > > > the
> > > > > > >> > affinity
> > > > > > >> > > as
> > > > > > >> > > > > >>>>>>>>>>>>> well
> > > > > > >> > > > > >>>>>>>>>>>>> as
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> anti-affinity of operators / partitions
> on
> > > > > > >> containers.
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> However, I noted the following points:
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> 1. AFAIK, we do not support
> > > affinity
> > > > > > >> > (locality)
> > > > > > >> > > > in
> > > > > > >> > > > > a
> > > > > > >> > > > > >>>>>>>>>>>>>>> general
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> sense.
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> The
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> affinity is only for a stream,
> > not
> > > > for
> > > > > > >> *any*
> > > > > > >> > two
> > > > > > >> > > > > >>>>>>>>>>>>> operators.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> So,
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> we
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> should also look at the
> general
> > > case
> > > > > and
> > > > > > >> see
> > > > > > >> > > how
> > > > > > >> > > > it
> > > > > > >> > > > > >>>>>>>>>>>>>>> can
> > > > > > >> > > > > >>>>>>>>>>>>>>> be
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> supported,
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> if
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> there are valid use cases.
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> 2. Coming to anti-affinity, we
> > > cannot
> > > > > > type
> > > > > > >> > cast
> > > > > > >> > > it
> > > > > > >> > > > > as
> > > > > > >> > > > > >>>>>>>>>>>>>> a
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> type of
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> affinity
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> rule. Saying "two operators
> must
> > > be
> > > > on
> > > > > > the
> > > > > > >> > same
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> container" is
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> very
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> different from saying "these
> two
> > > > > > operators
> > > > > > >> > must
> > > > > > >> > > > not
> > > > > > >> > > > > >>>>>>>>>>>>>>> be on
> > > > > > >> > > > > >>>>>>>>>>>>>>> the
> > > > > > >> > > > > >>>>>>>>>>>>>>> same
> > > > > > >> > > > > >>>>>>>>>>>>>>> container". In this case, the
> > > second
> > > > > one
> > > > > > >> is a
> > > > > > >> > > > much
> > > > > > >> > > > > >>>>>>>>>>>>>>> more
> > > > > > >> > > > > >>>>>>>>>>>>>>> relaxed
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> rule
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> as
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> compared to the first one.
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> 3. Once we have this generic set
> > of
> > > > > rules,
> > > > > > >> > there
> > > > > > >> > > > must
> > > > > > >> > > > > >>>>>>>>>>>>>> be a
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> "satisfiability" test run
> before
> > > > > > >> requesting
> > > > > > >> > > YARN
> > > > > > >> > > > > for
> > > > > > >> > > > > >>>>>>>>>>>>>>> containers.
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> If
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> the
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> request is not satisfiable,
> then
> > > > there
> > > > > is
> > > > > > >> no
> > > > > > >> > > point
> > > > > > >> > > > > >>>>>>>>>>>>> asking
> > > > > > >> > > > > >>>>>>>>>>>>> YARN
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> allocate
> > > > > > >> > > > > >>>>>>>>>>>>>>> containers in this manner. In
> > case
> > > > it
> > > > > is
> > > > > > >> not
> > > > > > >> > > > > >>>>>>>>>>>>>>> satisfiable, we
> > > > > > >> > > > > >>>>>>>>>>>>>>> must
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> also
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> have
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> a default order in which the
> > rules
> > > > can
> > > > > be
> > > > > > >> > > > "relaxed"
> > > > > > >> > > > > >>>>>>>>>>>>>> and
> > > > > > >> > > > > >>>>>>>>>>>>>> the
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> request
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> be
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> made
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> satisfiable. For example, some
> > very
> > > > > strict
> > > > > > >> > rules
> > > > > > >> > > > may
> > > > > > >> > > > > be
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> ignored,
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> or
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> made
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> less constraining ( for example
> > "on
> > > > the
> > > > > > >> same
> > > > > > >> > > > > >>>>>>>>>>>>> container" =>
> > > > > > >> > > > > >>>>>>>>>>>>> "on
> > > > > > >> > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>> the
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> same
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> node").
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> Thanks.
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> -Bhupesh
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:54 PM,
> > Aniruddha
> > > > > > >> Thombare <
> > > > > > >> > > > > >>>>>>>>>>>>>>> [email protected]> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> +1 On Chinmay's suggestion about
> having
> > > > single
> > > > > > >> set of
> > > > > > >> > > > > >>>>>>>>>>>>>>> affinity
> > > > > > >> > > > > >>>>>>>>>>>>>>> rules.
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> Thanks,
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> Aniruddha
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:57 PM,
> > Sandeep
> > > > > > >> Deshmukh <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> [email protected]
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> Between 2 operators, if one configures
> > > > > > >> > thread/container
> > > > > > >> > > > > local
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> and
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> anti-affinity as well, which one will
> > > take
> > > > > > >> affect?
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> The DAG validation step should error
> > out
> > > > in
> > > > > > this
> > > > > > >> > > case.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> +1 on suggestion by Chinmay to name
> it
> > > > > > "Affinity
> > > > > > >> > > Rules"
> > > > > > >> > > > > >>>>>>>>>>>>>>> than
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> anti-affinity. We are just extending
> our
> > > > > > container
> > > > > > >> > > > > allocation
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> scheme
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> support containers not to be
> allocated
> > > > > > together.
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> Regards,
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> Sandeep
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:43 PM,
> > Chinmay
> > > > > > >> Kolhatkar
> > > > > > >> > <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> [email protected]>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> Hi Isha,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> Couple of points:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> 1. About the interface to
> configuring
> > > > > > >> > anti-affinity,
> > > > > > >> > > as
> > > > > > >> > > > > per
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> suggestion
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> above there are 2 different way to
> > > > > configure
> > > > > > >> > > locality
> > > > > > >> > > > > and
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> anti-affinity:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> i.e. dag.setAttribute - for
> > anti-affinity
> > > > &
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> dag.addStream(...).setLocality
> > for
> > > > > > >> locality.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> Between 2 operators, if one
> configures
> > > > > > >> > > thread/container
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> local
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> anti-affinity as well, which one
> will
> > > > take
> > > > > > >> affect?
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> 2. Consider there could be such
> > > confusion
> > > > as
> > > > > > >> above,
> > > > > > >> > > > would
> > > > > > >> > > > > >>>>>>>>>>>>>>>> it
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> make
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> sense
> > > > > > >> > > > > >>>>>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> have a single API which takes care of
> > both
> > > > > > >> > > anti-affinity
> > > > > > >> > > > > and
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> locality.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> This
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> way, one is configurable.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> 3. This point is coming from how VM
> > > > affinity
> > > > > is
> > > > > > >> > > > configured
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> in
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> vSphere.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> The VMs are configured affinity are
> > > > called
> > > > > as
> > > > > > >> > > > "affinity
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> rules"
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> and
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> not
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> "anti-affinity rules". Ultimately
> idea
> > > is
> > > > to
> > > > > > >> > allocate
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> processing
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> nodes.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> Via "VM-VM affinity rules",
> > anti-affinity
> > > is
> > > > > > also
> > > > > > >> > > > > configured.
> > > > > > >> > > > > >>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>> But
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> there
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> is
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> a single set of rule definition for
> both
> > > > > > affinity
> > > > > > >> > > > (similar
> > > > > > >> > > > > to
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> locality
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> in
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> our case) and anti-affinity.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> Would it be a better approach for
> > > > configuring
> > > > > > >> > locality
> > > > > > >> > > > > rules
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> and
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> anti-affinity rules in a single
> rule
> > > and
> > > > > call
> > > > > > >> it
> > > > > > >> > > > > "affinity
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> rule".
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> Thanks,
> > > > > > >> > > > > >>>>>>>>>>>>>>> Chinmay.
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 12:24 PM, Yogi
> > > > > Devendra
> > > > > > <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> [email protected]
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> @Isha
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> I understand that anti-affinity
> across
> > > > > > >> application
> > > > > > >> > is
> > > > > > >> > > > not
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> straight-forward.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> It would be OK even if we do not
> > have
> > > it
> > > > > in
> > > > > > >> > > iteration
> > > > > > >> > > > > 1.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> But, for attributes syntax; I still
> > > think
> > > > > > that
> > > > > > >> > Java
> > > > > > >> > > > > object
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> should
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> be
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> avoided as they will be hard to
> > > configure
> > > > > > from
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> dt-site.xml or
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> other
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> config
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> files.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> Other suggestion for this could be
> JSON
> > > > > > >> > representation
> > > > > > >> > > > of
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> String
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> array:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> ["O2", "O3"]. (If operator names
> has
> > > > some
> > > > > > >> special
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> characters
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> like
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> "
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> or [
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> or , those will be escaped in the
> JSON
> > > > > > >> > > representation.)
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> Not sure if others agree on this; but
> > > > > attribute
> > > > > > >> > syntax
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> should
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> be
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> finalized
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> in iteration 1 to avoid backward
> > > > > > compatibility
> > > > > > >> > > issues
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> later.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> ~ Yogi
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> On 22 January 2016 at 00:43, Thomas
> > > Weise <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> [email protected]>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> Node based requests is the best
> > > approach
> > > > -
> > > > > if
> > > > > > >> it
> > > > > > >> > > works
> > > > > > >> > > > > :-)
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> Blacklisting will require to allocate
> > the
> > > > > > >> containers
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> sequentially.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> It
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> will
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> work, but slow down application
> > startup,
> > > > > > >> especially
> > > > > > >> > > for
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> larger
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> topologies.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jan 21, 2016 at 10:42 AM,
> > Isha
> > > > > > >> Arkatkar <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> [email protected]>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> Hi,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> Should we consider the node
> > > based
> > > > > > >> requests
> > > > > > >> > if
> > > > > > >> > > > it
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> works
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> with
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Capacity
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> Scheduler or avoid 2b approach
> > > > > altogether?
> > > > > > I
> > > > > > >> > > checked
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> node
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> requests
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> do
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> not work with fair scheduler on CDH
> > > > > cluster.
> > > > > > >> Yarn
> > > > > > >> > > does
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> not
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> return
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> any
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> container if hostname is given in
> the
> > > > > > container
> > > > > > >> > > > request.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> I
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> am
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> trying
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> setup a small virtual hortonworks
> > cluster
> > > > to
> > > > > > >> check
> > > > > > >> > the
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> this
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> behavior
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> on
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> that.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> YARN-2027 <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > https://issues.apache.org/jira/browse/YARN-2027
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> mentioned
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> that
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> container requests are not honored
> in
> > > > > > capacity
> > > > > > >> > > > scheduler
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> too.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> But I
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> am
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> not
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> sure if it is because of distro
> > > dependent
> > > > > > issue.
> > > > > > >> > > Please
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> share
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> insights.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> @Vlad, Adding support for regular
> > > > > > expression
> > > > > > >> > > sounds
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> good.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> We
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> could
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> translate to list of operator names
> > > > > > internally
> > > > > > >> > based
> > > > > > >> > > > on
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> regex.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> @Yogi, I went with a list of
> strings
> > > for
> > > > > > >> attribute
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> because
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> "O2,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> O3"
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> could
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> be a valid single operator name too
> :)
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> I am not sure of ways to implement
> > > > > > anti-affinity
> > > > > > >> > > across
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> application.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Though
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> something to consider for later
> > > > > iteration.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> Isha
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 8:59 PM,
> > > > Thomas
> > > > > > >> Weise <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> [email protected]>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> https://issues.apache.org/jira/browse/SLIDER-82
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 8:56 PM,
> > > Thomas
> > > > > > Weise
> > > > > > >> <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> [email protected]>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> The point was that containers
> are
> > > > taken
> > > > > > away
> > > > > > >> > from
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> other
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> apps
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> that
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> may
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> have
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> to discard work etc. It's not good
> > style
> > > > to
> > > > > > >> claim
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> resources
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> and
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> not
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> use
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> them eventually :-)
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> For this feature it is necessary
> to
> > > look
> > > > > at
> > > > > > >> the
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> scheduler
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> capabilities/semantics and
> > > > > limitations.
> > > > > > >> For
> > > > > > >> > > > > example,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> don't
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> bet
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> exclusively
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> on node requests if the goal is for
> > it
> > > to
> > > > > > work
> > > > > > >> > with
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> FairScheduler.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> Also look at Slider, which just
> > > recently
> > > > > > added
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> support
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> for
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> anti-affinity
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> (using node requests). When you
> run
> > > it
> > > > on
> > > > > > the
> > > > > > >> > CDH
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> cluster,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> it
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> probably
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> won't work...
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 3:19 PM,
> > Pramod
> > > > > > >> Immaneni <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> [email protected]
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Once released won't the
> > containers
> > > be
> > > > > > >> > available
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> again
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> in
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> pool.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> This
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> would only be optional and not
> > mandatory.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> Thanks
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> On Tue, Jan 19, 2016 at 2:02 PM,
> > > Thomas
> > > > > > >> Weise <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> [email protected]
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> How about also supporting a
> > minor
> > > > > > >> variation
> > > > > > >> > of
> > > > > > >> > > it
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> as
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> an
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> option
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> where it greedily gets the
> total
> > > > number
> > > > > > of
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> containers
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> and
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> discards
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ones
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> it
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> can't use and repeats the process
> > for
> > > > the
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> remaining
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> till
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> everything
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> has
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> been allocated.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> This is problematic as with
> resource
> > > > > > preemption
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> these
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> containers
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> will
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> be
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> potentially taken away from other
> > > > > > >> applications
> > > > > > >> > and
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> then
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> thrown
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> away.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Also does it make sense to
> > support
> > > > > > >> > anti-cluster
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> affinity?
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> Thanks
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jan 19, 2016 at 1:21
> > PM,
> > > > > Isha
> > > > > > >> > > Arkatkar
> > > > > > >> > > > <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> [email protected]>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> We want add support
> for
> > > > > > >> > Anti-affinity
> > > > > > >> > > in
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Apex
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> allow
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> applications
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> launch specific physical operators
> on
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> different
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> nodes(APEXCORE-10
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > https://issues.apache.org/jira/browse/APEXCORE-10
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ).
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> Want
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> request
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> your
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> suggestions/ideas for the same!
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> The reasons for using
> > > > > > anti-affinity
> > > > > > >> in
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> operators
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> could
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> be:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ensure
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> reliability, for performance
> reasons
> > > > (such
> > > > > as
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> application
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> may
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> not
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> want
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> 2
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> i/o intensive operators to land
> > on
> > > > the
> > > > > > same
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> node
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> improve
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> performance)
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> or
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> for some application specific
> > > > > constraints(for
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> example,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> 2
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> partitions
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> cannot
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> be run on the same node since
> > they
> > > > use
> > > > > > same
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> port
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> number).
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> This
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> is
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> general rationale for adding
> > > Anti-affinity
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> support.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Since, Yarn does not support
> > > > > > >> anti-affinity
> > > > > > >> > yet
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> (YARN-1042
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> <
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > https://issues.apache.org/jira/browse/YARN-1042
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> ),
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> we
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> need
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> to
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> implement
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> the logic in AM. Wanted to get
> your
> > > > views
> > > > > on
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> following
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> aspects
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> this
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> implementation:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> *1. How to specify anti-affinity
> > for
> > > > > > physical
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> operators/partitions
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> in
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> application:*
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> One way for this is
> to
> > > > have
> > > > > an
> > > > > > >> > > > attribute
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> setting
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> anti-affinity
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> at
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> the logical operator context.
> And
> > > an
> > > > > > >> operator
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> can
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> set
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> this
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> attribute
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> with
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> list of operator names which should
> > not
> > > > be
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> collocated.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Consider dag with 3
> > > > > operators:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> TestOperator o1 =
> > > > > > >> > > > > dag.addOperator("O1",
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator());
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> TestOperator o2 =
> > > > > > >> > dag.addOperator("O2",
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator());
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> TestOperator o3 =
> > > > > > >> dag.addOperator("O3",
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator());
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> To set anti-affinity for O1
> > > > operator:
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> dag.setAttribute(o1,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> OperatorContext.ANTI_AFFINITY,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>> new
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > ArrayList<String>(Arrays.asList("O2",
> > > > > > >> > > "O3")));
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> This would mean O1
> > > should
> > > > > not
> > > > > > >> be
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> allocated
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> on
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> nodes
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> containing
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> operators O2 and O3. This
> applies
> > > to
> > > > > all
> > > > > > >> > > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>> allocated
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> partitions
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> of
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> O1,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> O2,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>> O3.
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> Also, if same operator
> > > name
> > > > is
> > > > > > >> part
> > > > > > >> > of
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> anti-affinity
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> list,
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > > > > >>
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
