You could specify it by specifying the names of all operators that are part of the stream.
On Wed, Feb 24, 2016 at 3:44 PM, Amol Kekre <[email protected]> wrote: > These are two diff ways of specifying something similar, but not always > same. For example, how do I specify the following > > This stream has way to high throughput. Force all operators that connect to > it today (or in future) to be container local. > > With stream locality I get that as operator ports are added to the stream, > they automatically inherit stream locality. > > Thks, > Amol > > On Wed, Feb 24, 2016 at 3:37 PM, Pramod Immaneni <[email protected]> > wrote: > > > To explain the affinity support below seems to be a super-set that can > > consume the stream locality setting whether that was the original > intention > > or not. > > > > On Wed, Feb 24, 2016 at 3:35 PM, Pramod Immaneni <[email protected] > > > > wrote: > > > > > Why would we want two different ways of specifying the same thing? > > > > > > On Wed, Feb 24, 2016 at 3:29 PM, Amol Kekre <[email protected]> > > wrote: > > > > > >> Stream locality is a stream attribute. Affinity is operator attribute. > > >> Technically they may not be connected. Though operator affinity can be > > >> extended to specify a situation that implicity covers stream locality, > > it > > >> may be better to let user say the two following statements > > independently. > > >> > > >> - This stream is heavy on I/O and I want it to be ..._local // This > > >> statement is independent of future additions of operators to this > stream > > >> - I need these logical operators to not be (or be) on the same node > > >> > > >> Do note that container_local or node_local stream localities only > relate > > >> to > > >> connected operators. > > >> > > >> Thks > > >> Amol > > >> > > >> > > >> On Wed, Feb 24, 2016 at 1:54 PM, Pramod Immaneni < > > [email protected]> > > >> wrote: > > >> > > >> > Would it make sense to deprecate stream locality in favor of this? > > >> > > > >> > On Tue, Feb 23, 2016 at 6:05 PM, Isha Arkatkar < > [email protected]> > > >> > wrote: > > >> > > > >> > > Hi Pramod, > > >> > > > > >> > > We can have a list or operators or regex to specify operators > > list > > >> for > > >> > > affinity rule. Though, internal implementation will translate both > > >> these > > >> > > into list of pairs. Since, having pair of operators helps in > > >> validation > > >> > > phase. > > >> > > > > >> > > And yes, validation phase will catch conflicting locality or > > affinity > > >> > > rules and throw validation exception. > > >> > > > > >> > > Thanks for input regarding unifiers. It is probably alright to > not > > >> > include > > >> > > it in the affinity settings. Will leave the current implementation > > as > > >> is. > > >> > > > > >> > > Thanks! > > >> > > Isha > > >> > > > > >> > > > > >> > > On Tue, Feb 23, 2016 at 5:56 PM, Pramod Immaneni < > > >> [email protected] > > >> > > > > >> > > wrote: > > >> > > > > >> > > > Isha, > > >> > > > > > >> > > > If the implementation would support (eventually) affinity > between > > >> more > > >> > > than > > >> > > > 2 logical operators, instead of relying on specifying it as > > multiple > > >> > > rules > > >> > > > why not make provision in the API to specify more than 2 logical > > >> > > operators? > > >> > > > If there is a stream locality rule that conflicts with the > > affinity > > >> > rule > > >> > > > will the validation catch this? With unifiers the common case of > > MxN > > >> > > > locality is already chosen based on downstream operator. In > other > > >> cases > > >> > > > such as single downstream or cascade case I am not sure it is > all > > >> the > > >> > > > important to have it. > > >> > > > > > >> > > > Thanks > > >> > > > > > >> > > > On Tue, Feb 23, 2016 at 5:37 PM, Isha Arkatkar < > > >> [email protected]> > > >> > > > wrote: > > >> > > > > > >> > > > > Hi all, > > >> > > > > > > >> > > > > I have opened a review only pull request for handling > > affinity > > >> > > rules > > >> > > > in > > >> > > > > Apex: > > >> > > > > https://github.com/apache/incubator-apex-core/pull/234 > > >> > > > > > > >> > > > > Just wanted to confirm with everyone, if the APIs for > > specifying > > >> > > > affinity > > >> > > > > rules look alright. Also, wanted to get views on how unifiers > > >> should > > >> > be > > >> > > > > handled in case of operator affinity rules. More details > follow: > > >> > > > > > > >> > > > > I have added a list of affinity rules as an attribute in Dag > > >> > context. > > >> > > > For > > >> > > > > now, have added AffinityRule to be specified for operator > pairs. > > >> We > > >> > can > > >> > > > add > > >> > > > > regex support in next iteration. > > >> > > > > > > >> > > > > Here is sample usage: > > >> > > > > > > >> > > > > List<AffinityRule> rules = new ArrayList<>(); > > >> > > > > // To add node locality between 2 not connected operators > > rand > > >> > and > > >> > > > > console: > > >> > > > > rules.add(new AffinityRule(Type.AFFINITY, new OperatorPair( > > >> > "rand", > > >> > > > > "console"), Locality.NODE_LOCAL, false)); > > >> > > > > > > >> > > > > // To add anti-affinity between all partitions of rand > > operator > > >> > with > > >> > > > all > > >> > > > > partitions of passThru operator: > > >> > > > > rules.add(new AffinityRule(Type.ANTI_AFFINITY, new > > >> OperatorPair( > > >> > > > "rand", > > >> > > > > "passThru"), Locality.NODE_LOCAL, false)); > > >> > > > > > > >> > > > > // To add anti-affinity between partitions of passThru > > >> operator, > > >> > > give > > >> > > > > same operator name in pair: > > >> > > > > rules.add(new AffinityRule(Type.ANTI_AFFINITY, new > > >> OperatorPair( > > >> > > > > "passThru", "passThru"), Locality.NODE_LOCAL, false)); > > >> > > > > > > >> > > > > // Set the rules in dag context > > >> > > > > dag.setAttribute(DAGContext.AFFINITY_RULES, rules); > > >> > > > > > > >> > > > > > > >> > > > > Please find a sample application using these affinity rules > > here: > > >> > > > > > > >> https://github.com/ishark/Apex-Samples/tree/master/affinity-example > > >> > > > > > > >> > > > > > > >> > > > > The actual implementation of affinity rules heavily depends > node > > >> > > specific > > >> > > > > requests, that was already being handled in > > >> > StreamingAppMasterService. > > >> > > To > > >> > > > > handle node requests for cloudera, I have added an override to > > >> > > blacklist > > >> > > > > all other nodes except the ones where container request is to > be > > >> > > issued. > > >> > > > > > > >> > > > > There is one open question I wanted to bring up: In case of > > >> > partitioned > > >> > > > > operators, should the unifiers also follow affinity or > > >> anti-affinity > > >> > > > rules? > > >> > > > > Or should they be independent. For now, I have kept them > > >> independent > > >> > > and > > >> > > > > only the actual operators follow affinity rules. Please > suggest > > >> what > > >> > > > would > > >> > > > > make more sense from user's specification perspective. > > >> > > > > > > >> > > > > Thanks, > > >> > > > > Isha > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > On Mon, Feb 1, 2016 at 2:12 PM, Isha Arkatkar < > > >> [email protected]> > > >> > > > > wrote: > > >> > > > > > > >> > > > > > Hi folks, > > >> > > > > > > > >> > > > > > Summarizing the proposal for affinity/anti-affinity rules > > in > > >> > apex > > >> > > as > > >> > > > > > per discussions in this mail thread. Please suggest if I > > missed > > >> > > > > something. > > >> > > > > > > > >> > > > > > * For configuration:* > > >> > > > > > > > >> > > > > > - We will have application level affinity/anti-affinity > > rules. > > >> > For > > >> > > > the > > >> > > > > > first iteration, we can support for (anti-)affinity among > > >> operators > > >> > > > > within > > >> > > > > > single application. We can revisit it in the next iteration > to > > >> > > support > > >> > > > > > across applications. > > >> > > > > > > > >> > > > > > - Each rule will consist of 4 components: > > >> > > > > > <List of operators>, AFFINITY/ANTI-AFFINITY, > > >> STRICT/RELAXED > > >> > > > > policy, > > >> > > > > > CONTAINER/NODE/RACK > > >> > > > > > I have checked that Apex supports container locality > between 2 > > >> > > > operators > > >> > > > > > that are not connected by Steam. Though did not find an API > > for > > >> > > setting > > >> > > > > it > > >> > > > > > through application. So, affinity rules can cover container > > >> > affinity > > >> > > as > > >> > > > > > well. > > >> > > > > > We will not support THREAD locality in app level affinity or > > >> > > > > > anti-affinity, since it does not apply to unconnected > > operators. > > >> > > > > > > > >> > > > > > The list of operators can also be specified by regex > > >> > > > > > > > >> > > > > > *For Implementation: * > > >> > > > > > > > >> > > > > > - I have verified that Node specific requests are honored > by > > >> > > > > HortonWorks > > >> > > > > > for both Fair and Capacity scheduler. So we can go with the > > >> > approach > > >> > > > of > > >> > > > > > node specific container requests to implement affinity > rules. > > >> > > > > > > > >> > > > > > - However, we may need to fall back on blacklist approach > for > > >> CDH > > >> > > > > distro, > > >> > > > > > as node specific requests may not work well. As: > > >> > > > > > -> With Fair scheduler, node specific requests are > > not > > >> > > honored > > >> > > > > in > > >> > > > > > CDH 4. Application master keeps sending requests to Yarn, > but > > no > > >> > > > > containers > > >> > > > > > are allocated. > > >> > > > > > -> I could not yet verify the same with Fair and > > >> Capacity > > >> > > > > > Scheduler on CDH 5. Ran into some configuration issues when > > >> setting > > >> > > up > > >> > > > > CDH > > >> > > > > > 5.5.1. Will update the thread with findings. > > >> > > > > > > > >> > > > > > I will try to build a prototype for affinity rules with > > these > > >> > > things > > >> > > > in > > >> > > > > > mind. Please let me know if anyone would like to collaborate > > on > > >> > this > > >> > > > > > effort! :) > > >> > > > > > > > >> > > > > > Thanks! > > >> > > > > > Isha > > >> > > > > > > > >> > > > > > On Mon, Jan 25, 2016 at 10:23 AM, Vlad Rozov < > > >> > > [email protected]> > > >> > > > > > wrote: > > >> > > > > > > > >> > > > > >> Yes, with ability to refer not only to a host name, but to > > >> > operators > > >> > > > > >> (possibly both in the same and other applications). In > > general > > >> > this > > >> > > > > should > > >> > > > > >> allow to specify both affinity and anti affinity using > > >> > expressions. > > >> > > > For > > >> > > > > >> example LOCALITY_CONTAINER = "{app1.op1 or app1.op2} and > > >> > > {!app2.op1}". > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> On 1/25/16 09:39, Thomas Weise wrote: > > >> > > > > >> > > >> > > > > >>> Agreed. Originally I thought you wanted both operators on > > the > > >> > same > > >> > > > > host. > > >> > > > > >>> To > > >> > > > > >>> pin them to the same container the same idea can be > applied, > > >> > > through > > >> > > > a > > >> > > > > >>> "LOCALITY_CONTAINER" attribute. > > >> > > > > >>> > > >> > > > > >>> > > >> > > > > >>> On Mon, Jan 25, 2016 at 9:18 AM, Vlad Rozov < > > >> > > [email protected] > > >> > > > > > > >> > > > > >>> wrote: > > >> > > > > >>> > > >> > > > > >>> Works as a workaround and requires two extra ports that > > >> operators > > >> > > > > >>>> designer > > >> > > > > >>>> may not necessarily provide out of the box. We may apply > > the > > >> > same > > >> > > > hack > > >> > > > > >>>> to > > >> > > > > >>>> anti-affinity and request that any two operators with > > >> > > anti-affinity > > >> > > > > rule > > >> > > > > >>>> are connected by non functional stream, but it does not > > look > > >> > like > > >> > > a > > >> > > > > good > > >> > > > > >>>> option :). > > >> > > > > >>>> > > >> > > > > >>>> > > >> > > > > >>>> > > >> > > > > >>>> On 1/25/16 08:52, Thomas Weise wrote: > > >> > > > > >>>> > > >> > > > > >>>> Sandeep suggested to connect them with a stream. > > >> > > > > >>>>> > > >> > > > > >>>>> On Mon, Jan 25, 2016 at 8:49 AM, Vlad Rozov < > > >> > > > [email protected] > > >> > > > > > > > >> > > > > >>>>> wrote: > > >> > > > > >>>>> > > >> > > > > >>>>> Sorry, possibly I miss something. X and Z are not > directly > > >> > > > connected > > >> > > > > >>>>> by a > > >> > > > > >>>>> > > >> > > > > >>>>>> stream, how can I apply THREAD_LOCAL? > > >> > > > > >>>>>> > > >> > > > > >>>>>> > > >> > > > > >>>>>> On 1/25/16 08:44, Thomas Weise wrote: > > >> > > > > >>>>>> > > >> > > > > >>>>>> Why does THREAD_LOCAL not work? > > >> > > > > >>>>>> > > >> > > > > >>>>>>> On Mon, Jan 25, 2016 at 7:43 AM, Vlad Rozov < > > >> > > > > [email protected] > > >> > > > > >>>>>>> > > > >> > > > > >>>>>>> wrote: > > >> > > > > >>>>>>> > > >> > > > > >>>>>>> Neither LOCALITY_HOST or THREAD_LOCAL provides > required > > >> > > > > >>>>>>> functionality. > > >> > > > > >>>>>>> The > > >> > > > > >>>>>>> > > >> > > > > >>>>>>> goal is to deploy X and Z to the same container/JVM. X > > >> may be > > >> > > > > >>>>>>>> deployed > > >> > > > > >>>>>>>> on > > >> > > > > >>>>>>>> any node. Z needs to be deployed on the same > container > > >> as X, > > >> > > so > > >> > > > X > > >> > > > > >>>>>>>> does > > >> > > > > >>>>>>>> not > > >> > > > > >>>>>>>> have any host affinity, and Z has affinity to X, not > to > > >> any > > >> > > > > specific > > >> > > > > >>>>>>>> host. > > >> > > > > >>>>>>>> If database client caches writes, reading inside the > > same > > >> > JVM > > >> > > > will > > >> > > > > >>>>>>>> be > > >> > > > > >>>>>>>> in > > >> > > > > >>>>>>>> process lookup, while deploying Z to any other > host/JVM > > >> > leads > > >> > > to > > >> > > > > >>>>>>>> inter > > >> > > > > >>>>>>>> process or inter host lookup. > > >> > > > > >>>>>>>> > > >> > > > > >>>>>>>> > > >> > > > > >>>>>>>> On 1/24/16 21:03, Thomas Weise wrote: > > >> > > > > >>>>>>>> > > >> > > > > >>>>>>>> There are the attributes LOCALITY_HOST and > > LOCALITY_RACK > > >> for > > >> > > an > > >> > > > > >>>>>>>> operator > > >> > > > > >>>>>>>> > > >> > > > > >>>>>>>> to > > >> > > > > >>>>>>>>> achieve precisely what you are describing. The value > > >> would > > >> > be > > >> > > > an > > >> > > > > >>>>>>>>> alias > > >> > > > > >>>>>>>>> that > > >> > > > > >>>>>>>>> can be mapped to physical hosts for allocation. The > > same > > >> > > could > > >> > > > be > > >> > > > > >>>>>>>>> used > > >> > > > > >>>>>>>>> of > > >> > > > > >>>>>>>>> anti-affinity, by assigning different values. > > >> > > > > >>>>>>>>> > > >> > > > > >>>>>>>>> Stream locality is a special case of expressing > > affinity > > >> > for > > >> > > 2 > > >> > > > > >>>>>>>>> operators. > > >> > > > > >>>>>>>>> Note that we also need a way to extend this to > > >> partitions > > >> > of > > >> > > > the > > >> > > > > >>>>>>>>> same > > >> > > > > >>>>>>>>> operator. > > >> > > > > >>>>>>>>> > > >> > > > > >>>>>>>>> > > >> > > > > >>>>>>>>> On Sun, Jan 24, 2016 at 8:52 PM, Vlad Rozov < > > >> > > > > >>>>>>>>> [email protected]> > > >> > > > > >>>>>>>>> wrote: > > >> > > > > >>>>>>>>> > > >> > > > > >>>>>>>>> It may be quite extreme use case, but suppose that > > >> > operator X > > >> > > > > >>>>>>>>> writes > > >> > > > > >>>>>>>>> to > > >> > > > > >>>>>>>>> a > > >> > > > > >>>>>>>>> > > >> > > > > >>>>>>>>> distributed in-memory database and operator Z reads > > from > > >> > it. > > >> > > Z > > >> > > > is > > >> > > > > >>>>>>>>> not > > >> > > > > >>>>>>>>> > > >> > > > > >>>>>>>>>> directly connected to X. In such cases it may be > > >> necessary > > >> > > to > > >> > > > > let > > >> > > > > >>>>>>>>>> an > > >> > > > > >>>>>>>>>> application request container affinity (deploying > to > > >> the > > >> > > same > > >> > > > > JVM) > > >> > > > > >>>>>>>>>> for > > >> > > > > >>>>>>>>>> operators X and Z as writes may be cached and reads > > >> from > > >> > the > > >> > > > > same > > >> > > > > >>>>>>>>>> JVM > > >> > > > > >>>>>>>>>> will > > >> > > > > >>>>>>>>>> be potentially faster. > > >> > > > > >>>>>>>>>> > > >> > > > > >>>>>>>>>> Technically the same may be applied to thread local > > on > > >> > NUMA > > >> > > > > >>>>>>>>>> boxes, as > > >> > > > > >>>>>>>>>> NUMA > > >> > > > > >>>>>>>>>> aware deployment of containers is not supported. > > >> > > > > >>>>>>>>>> > > >> > > > > >>>>>>>>>> Vlad > > >> > > > > >>>>>>>>>> > > >> > > > > >>>>>>>>>> On 1/23/16 18:01, Yogi Devendra wrote: > > >> > > > > >>>>>>>>>> > > >> > > > > >>>>>>>>>> @Isha > > >> > > > > >>>>>>>>>> > > >> > > > > >>>>>>>>>> In my opinion: > > >> > > > > >>>>>>>>>> > > >> > > > > >>>>>>>>>>> THREAD_LOCAL, CONTAINER_LOCAL on stream is a > special > > >> case > > >> > > of > > >> > > > > >>>>>>>>>>> generic > > >> > > > > >>>>>>>>>>> rules > > >> > > > > >>>>>>>>>>> for Operator X and Operator Y. > > >> > > > > >>>>>>>>>>> > > >> > > > > >>>>>>>>>>> We can say that, THREAD_LOCAL, CONTAINER_LOCAL > would > > >> be > > >> > > > > >>>>>>>>>>> applicable > > >> > > > > >>>>>>>>>>> only > > >> > > > > >>>>>>>>>>> if > > >> > > > > >>>>>>>>>>> operator X and Y are connected by stream. But, way > > to > > >> > > express > > >> > > > > >>>>>>>>>>> this > > >> > > > > >>>>>>>>>>> should > > >> > > > > >>>>>>>>>>> be similar to other rules for affinity. > > >> > > > > >>>>>>>>>>> > > >> > > > > >>>>>>>>>>> ~ Yogi > > >> > > > > >>>>>>>>>>> > > >> > > > > >>>>>>>>>>> On 24 January 2016 at 03:49, Isha Arkatkar < > > >> > > > > [email protected] > > >> > > > > >>>>>>>>>>> > > > >> > > > > >>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>> > > >> > > > > >>>>>>>>>>> Hey Chinmay, > > >> > > > > >>>>>>>>>>> > > >> > > > > >>>>>>>>>>> I certainly agree on common set of rules > for > > >> > > > > configuring > > >> > > > > >>>>>>>>>>> > > >> > > > > >>>>>>>>>>> affinity! > > >> > > > > >>>>>>>>>>>> Well > > >> > > > > >>>>>>>>>>>> put by a concrete example. :) > > >> > > > > >>>>>>>>>>>> Only thing I would like to point is: > > >> affinity of > > >> > > > > >>>>>>>>>>>> operators > > >> > > > > >>>>>>>>>>>> should > > >> > > > > >>>>>>>>>>>> not > > >> > > > > >>>>>>>>>>>> cover thread and container locality. Since this > is > > >> only > > >> > > > > >>>>>>>>>>>> applicable > > >> > > > > >>>>>>>>>>>> when > > >> > > > > >>>>>>>>>>>> operators are connected by stream. So, it makes > > >> sense to > > >> > > > have > > >> > > > > >>>>>>>>>>>> it on > > >> > > > > >>>>>>>>>>>> Stream > > >> > > > > >>>>>>>>>>>> rather than in common configuration. > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> And yes, DAG.validate should only check > for > > >> > > REQUIRED > > >> > > > or > > >> > > > > >>>>>>>>>>>> STRICT > > >> > > > > >>>>>>>>>>>> policy. We > > >> > > > > >>>>>>>>>>>> can agree on one of the terminologies > > STRICT/RELAXED > > >> or > > >> > > > > >>>>>>>>>>>> REQUIRED/PREFERRED. > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> Thanks! > > >> > > > > >>>>>>>>>>>> Isha > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> On Fri, Jan 22, 2016 at 8:38 PM, Chinmay > Kolhatkar > > < > > >> > > > > >>>>>>>>>>>> [email protected]> > > >> > > > > >>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> Hi Isha, Bhupesh, > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> When I suggested singe affinity rule, I was > mainly > > >> > talking > > >> > > > > about > > >> > > > > >>>>>>>>>>>> "how > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> to" > > >> > > > > >>>>>>>>>>>>> of configuration and not of implementation. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> I see locality is in a way suggesting an > affinity > > of > > >> > > > > operators. > > >> > > > > >>>>>>>>>>>>> They're > > >> > > > > >>>>>>>>>>>>> close terminologies. > > >> > > > > >>>>>>>>>>>>> By configuring a locality on stream, we're also, > > in > > >> a > > >> > > way, > > >> > > > > >>>>>>>>>>>>> defining > > >> > > > > >>>>>>>>>>>>> affinity of operators. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> Until now, only locality was there and hence was > > >> > straight > > >> > > > > >>>>>>>>>>>>> forward > > >> > > > > >>>>>>>>>>>>> in > > >> > > > > >>>>>>>>>>>>> configuration for user. > > >> > > > > >>>>>>>>>>>>> Tomorrow, when anti-affinity configuration comes > > up, > > >> > one > > >> > > > > might > > >> > > > > >>>>>>>>>>>>> get > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> confused > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> on how to best use both locality & > anti-affinity. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> Hence suggested to make both (locality/affinity & > > >> > > > > >>>>>>>>>>>> anti-affinity) as > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> a > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> part > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> of single configuration. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> Suggestion is to have a more commonly adopted > > >> > > configuration > > >> > > > > >>>>>>>>>>>> which > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> admins > > >> > > > > >>>>>>>>>>>>> and developer's are familiar with. > > >> > > > > >>>>>>>>>>>>> Again referring to vShere Hypervisor's affinity > > >> rules. > > >> > I > > >> > > > > think > > >> > > > > >>>>>>>>>>>>> they > > >> > > > > >>>>>>>>>>>>> have > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> a > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> single configuration which does both. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> Having said that, here is a quick suggestion on > how > > >> both > > >> > > can > > >> > > > > be > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> achieved > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> in > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> a single configuration: > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> CATEGORY TYPE POLICY > > >> ENTITIES > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> Affinity THREAD REQUIRED > > >> O1, O2 > > >> > > > > >>>>>>>>>>>>> //Meaning > > >> > > > > >>>>>>>>>>>>> Operator1 & Operator2 should be thread local > > >> > > > > >>>>>>>>>>>>> Affinity NODE PREFERRED > > >> O3, O4 > > >> > > > > >>>>>>>>>>>>> //Meaning > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> O3 & > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> O4 are preferred to be in node local > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> AntiAffinity NODE REQUIRED > > O1, > > >> O4 > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> //Meaning > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> O1 & > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> O4 should not be on same node. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> AntiAffinity RACK PREFERRED > O2, > > O4 > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> //Meaning > > >> > > > > >>>>>>>>>>>>> O2 > > >> > > > > >>>>>>>>>>>>> & > > >> > > > > >>>>>>>>>>>>> O4 are preferred not to be on same rack. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> Linux setting affinity of CPUs for threads is > > >> another > > >> > way > > >> > > > of > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> configuration > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> we can take a look at. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> Learning from these commonly adopted > configuration > > >> > > pattern, > > >> > > > we > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> should > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> come > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> up with best configuration suitable for > > distributed > > >> > > > > >>>>>>>>>>>>> environment. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> Idea here is to not have our own configuration > and > > >> give > > >> > > > > >>>>>>>>>>>> something > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> new > > >> > > > > >>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>> the users. Otherwise such an important concept > > might > > >> > > > quickly > > >> > > > > >>>>>>>>>>>>> get > > >> > > > > >>>>>>>>>>>>> lost. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> Regarding the DAG.validate I think we would need > > to > > >> add > > >> > > > some > > >> > > > > >>>>>>>>>>>>> new > > >> > > > > >>>>>>>>>>>>> stuff > > >> > > > > >>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>> take care of anti-affinity. > > >> > > > > >>>>>>>>>>>>> Plus, anti-affinity/affinity should be validated > > at > > >> > > > > >>>>>>>>>>>>> DAG.validate > > >> > > > > >>>>>>>>>>>>> only > > >> > > > > >>>>>>>>>>>>> for > > >> > > > > >>>>>>>>>>>>> the ones which are required. > > >> > > > > >>>>>>>>>>>>> For preferred policies, validation in logical > plan > > >> > might > > >> > > > be a > > >> > > > > >>>>>>>>>>>>> early > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> check. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> Thanks, > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>> Chinmay. > > >> > > > > >>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> On Sat, Jan 23, 2016 at 3:15 AM, Isha Arkatkar < > > >> > > > > >>>>>>>>>>>>> [email protected]> > > >> > > > > >>>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> Hi, > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> Thanks for inputs! I like the idea of > > having > > >> > > single > > >> > > > > set > > >> > > > > >>>>>>>>>>>>> of > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> (anti?)affinity rules at dag context level. > > >> > > > > >>>>>>>>>>>>>> There could still be conflicts based on > > >> > Locality > > >> > > > for > > >> > > > > >>>>>>>>>>>>>> Streams > > >> > > > > >>>>>>>>>>>>>> or > > >> > > > > >>>>>>>>>>>>>> Node > > >> > > > > >>>>>>>>>>>>>> locality attribute set for operators. But as > > >> Sandeep > > >> > > > > >>>>>>>>>>>>>> suggested, I > > >> > > > > >>>>>>>>>>>>>> also > > >> > > > > >>>>>>>>>>>>>> think Dag.validate should fail in case of > > >> > contradicting > > >> > > > > >>>>>>>>>>>>>> constraints. > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> We currently do not have 'affinity' among > > non > > >> > > stream > > >> > > > > >>>>>>>>>>>>>> operators, as > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> Bhupesh > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> pointed out. It is somewhat achievable by > > >> requesting > > >> > > node > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> locality > > >> > > > > >>>>>>>>>>>>> for > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> operators (Assuming node requests work as > > expected). > > >> > But > > >> > > > > >>>>>>>>>>>>> should we > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> consider > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> adding affinity specifications support as well > > >> along > > >> > > with > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> anti-affinity? > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> Regarding specification of attributes > from > > >> > > > > dt-site.xml. > > >> > > > > >>>>>>>>>>>>>> We > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> can > > >> > > > > >>>>>>>>>>>>> go > > >> > > > > >>>>>>>>>>>>> with > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> Json like string or even xml representation for > > >> complex > > >> > > > > >>>>>>>>>>>>> objects. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> What > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> is > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> our current behavior for setting Java object > > >> > properties > > >> > > > > >>>>>>>>>>>>>> through > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> XML? > > >> > > > > >>>>>>>>>>>>> We > > >> > > > > >>>>>>>>>>>>> can > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> follow the same for this as well. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> As for precedence or ability to satisfy > > >> > > constraints: > > >> > > > > >>>>>>>>>>>>> Right > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> now in > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> normal > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> scenario, if resources are not available for > > >> > allocating > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> containers, > > >> > > > > >>>>>>>>>>>>> we > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> keep > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> sending to request till all are obtained. > > >> Likewise, in > > >> > > > case > > >> > > > > of > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> strict > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> anti-affinity policy, we should keep the > > >> application in > > >> > > > > >>>>>>>>>>>>> ACCEPTED > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> state > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> till > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> the anti-affinity constraint is satisfied. For > > >> relaxed > > >> > > > > >>>>>>>>>>>>>> policy, we > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> can > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> decide timeout for relaxing the anti-affinity > > rule. > > >> > > Please > > >> > > > > note > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> this > > >> > > > > >>>>>>>>>>>>>> applies only when we have non-contradicting > > rules. > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> Thanks! > > >> > > > > >>>>>>>>>>>>>> Isha > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 5:30 AM, Bhupesh > Chawda < > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> [email protected] > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> I agree on having a single set of rules for the > > >> > affinity > > >> > > as > > >> > > > > >>>>>>>>>>>>> well > > >> > > > > >>>>>>>>>>>>> as > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> anti-affinity of operators / partitions on > > >> containers. > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> However, I noted the following points: > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> 1. AFAIK, we do not support affinity > > >> > (locality) > > >> > > > in > > >> > > > > a > > >> > > > > >>>>>>>>>>>>>>> general > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> sense. > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> The > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> affinity is only for a stream, not for > > >> *any* > > >> > two > > >> > > > > >>>>>>>>>>>>> operators. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> So, > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> we > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> should also look at the general case > and > > >> see > > >> > > how > > >> > > > it > > >> > > > > >>>>>>>>>>>>>>> can > > >> > > > > >>>>>>>>>>>>>>> be > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> supported, > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> if > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> there are valid use cases. > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> 2. Coming to anti-affinity, we cannot > > type > > >> > cast > > >> > > it > > >> > > > > as > > >> > > > > >>>>>>>>>>>>>> a > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> type of > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> affinity > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> rule. Saying "two operators must be on > > the > > >> > same > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> container" is > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> very > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> different from saying "these two > > operators > > >> > must > > >> > > > not > > >> > > > > >>>>>>>>>>>>>>> be on > > >> > > > > >>>>>>>>>>>>>>> the > > >> > > > > >>>>>>>>>>>>>>> same > > >> > > > > >>>>>>>>>>>>>>> container". In this case, the second > one > > >> is a > > >> > > > much > > >> > > > > >>>>>>>>>>>>>>> more > > >> > > > > >>>>>>>>>>>>>>> relaxed > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> rule > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> as > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> compared to the first one. > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> 3. Once we have this generic set of > rules, > > >> > there > > >> > > > must > > >> > > > > >>>>>>>>>>>>>> be a > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> "satisfiability" test run before > > >> requesting > > >> > > YARN > > >> > > > > for > > >> > > > > >>>>>>>>>>>>>>> containers. > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> If > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> the > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> request is not satisfiable, then there > is > > >> no > > >> > > point > > >> > > > > >>>>>>>>>>>>> asking > > >> > > > > >>>>>>>>>>>>> YARN > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> allocate > > >> > > > > >>>>>>>>>>>>>>> containers in this manner. In case it > is > > >> not > > >> > > > > >>>>>>>>>>>>>>> satisfiable, we > > >> > > > > >>>>>>>>>>>>>>> must > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> also > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> have > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> a default order in which the rules can > be > > >> > > > "relaxed" > > >> > > > > >>>>>>>>>>>>>> and > > >> > > > > >>>>>>>>>>>>>> the > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> request > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> be > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> made > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> satisfiable. For example, some very > strict > > >> > rules > > >> > > > may > > >> > > > > be > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> ignored, > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> or > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> made > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> less constraining ( for example "on the > > >> same > > >> > > > > >>>>>>>>>>>>> container" => > > >> > > > > >>>>>>>>>>>>> "on > > >> > > > > >>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>> the > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> same > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> node"). > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> Thanks. > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> -Bhupesh > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:54 PM, Aniruddha > > >> Thombare < > > >> > > > > >>>>>>>>>>>>>>> [email protected]> wrote: > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> +1 On Chinmay's suggestion about having single > > >> set of > > >> > > > > >>>>>>>>>>>>>>> affinity > > >> > > > > >>>>>>>>>>>>>>> rules. > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> Thanks, > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> Aniruddha > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:57 PM, Sandeep > > >> Deshmukh < > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> [email protected] > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> Between 2 operators, if one configures > > >> > thread/container > > >> > > > > local > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> and > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> anti-affinity as well, which one will take > > >> affect? > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> The DAG validation step should error out in > > this > > >> > > case. > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> +1 on suggestion by Chinmay to name it > > "Affinity > > >> > > Rules" > > >> > > > > >>>>>>>>>>>>>>> than > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> anti-affinity. We are just extending our > > container > > >> > > > > allocation > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> scheme > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> support containers not to be allocated > > together. > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> Regards, > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> Sandeep > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:43 PM, Chinmay > > >> Kolhatkar > > >> > < > > >> > > > > >>>>>>>>>>>>>>>>> [email protected]> > > >> > > > > >>>>>>>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> Hi Isha, > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> Couple of points: > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> 1. About the interface to configuring > > >> > anti-affinity, > > >> > > as > > >> > > > > per > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> suggestion > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> above there are 2 different way to > configure > > >> > > locality > > >> > > > > and > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> anti-affinity: > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> i.e. dag.setAttribute - for anti-affinity & > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> dag.addStream(...).setLocality for > > >> locality. > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> Between 2 operators, if one configures > > >> > > thread/container > > >> > > > > >>>>>>>>>>>>>>>>>> local > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> and > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> anti-affinity as well, which one will take > > >> affect? > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> 2. Consider there could be such confusion as > > >> above, > > >> > > > would > > >> > > > > >>>>>>>>>>>>>>>> it > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> make > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> sense > > >> > > > > >>>>>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> have a single API which takes care of both > > >> > > anti-affinity > > >> > > > > and > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> locality. > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> This > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> way, one is configurable. > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> 3. This point is coming from how VM affinity > is > > >> > > > configured > > >> > > > > >>>>>>>>>>>>>>>>> in > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> vSphere. > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> The VMs are configured affinity are called > as > > >> > > > "affinity > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> rules" > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> and > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> not > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> "anti-affinity rules". Ultimately idea is to > > >> > allocate > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> processing > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> nodes. > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> Via "VM-VM affinity rules", anti-affinity is > > also > > >> > > > > configured. > > >> > > > > >>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>> But > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> there > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> is > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> a single set of rule definition for both > > affinity > > >> > > > (similar > > >> > > > > to > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> locality > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> in > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> our case) and anti-affinity. > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> Would it be a better approach for configuring > > >> > locality > > >> > > > > rules > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> and > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> anti-affinity rules in a single rule and > call > > >> it > > >> > > > > "affinity > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> rule". > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> Thanks, > > >> > > > > >>>>>>>>>>>>>>> Chinmay. > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 12:24 PM, Yogi > Devendra > > < > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> [email protected] > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> @Isha > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> I understand that anti-affinity across > > >> application > > >> > is > > >> > > > not > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> straight-forward. > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> It would be OK even if we do not have it > in > > >> > > iteration > > >> > > > > 1. > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> But, for attributes syntax; I still think > > that > > >> > Java > > >> > > > > object > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> should > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> be > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> avoided as they will be hard to configure > > from > > >> > > > > >>>>>>>>>>>>>>>>> dt-site.xml or > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> other > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> config > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> files. > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> Other suggestion for this could be JSON > > >> > representation > > >> > > > of > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> String > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> array: > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> ["O2", "O3"]. (If operator names has some > > >> special > > >> > > > > >>>>>>>>>>>>>>>>> characters > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> like > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> " > > >> > > > > >>>>>>>>>>>>>>>>> or [ > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> or , those will be escaped in the JSON > > >> > > representation.) > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> Not sure if others agree on this; but > attribute > > >> > syntax > > >> > > > > >>>>>>>>>>>>>>>>> should > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> be > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> finalized > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> in iteration 1 to avoid backward > > compatibility > > >> > > issues > > >> > > > > >>>>>>>>>>>>>>>>> later. > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> ~ Yogi > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> On 22 January 2016 at 00:43, Thomas Weise < > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> [email protected]> > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> Node based requests is the best approach - > if > > >> it > > >> > > works > > >> > > > > :-) > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> Blacklisting will require to allocate the > > >> containers > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> sequentially. > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> It > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> will > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> work, but slow down application startup, > > >> especially > > >> > > for > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> larger > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> topologies. > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jan 21, 2016 at 10:42 AM, Isha > > >> Arkatkar < > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> [email protected]> > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> Hi, > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> Should we consider the node based > > >> requests > > >> > if > > >> > > > it > > >> > > > > >>>>>>>>>>>>>>>>>>> works > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> with > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Capacity > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> Scheduler or avoid 2b approach > altogether? > > I > > >> > > checked > > >> > > > > >>>>>>>>>>>>>>>>>>> that > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> node > > >> > > > > >>>>>>>>>>>>>>>>> requests > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> do > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> not work with fair scheduler on CDH > cluster. > > >> Yarn > > >> > > does > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> not > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> return > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> any > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> container if hostname is given in the > > container > > >> > > > request. > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> I > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> am > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> trying > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> setup a small virtual hortonworks cluster to > > >> check > > >> > the > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> this > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> behavior > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> on > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> that. > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> YARN-2027 < > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> > > https://issues.apache.org/jira/browse/YARN-2027 > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> mentioned > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> that > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> container requests are not honored in > > capacity > > >> > > > scheduler > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> too. > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> But I > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> am > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> not > > >> > > > > >>>>>>>>>>>>>>>>> sure if it is because of distro dependent > > issue. > > >> > > Please > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> share > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> insights. > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> @Vlad, Adding support for regular > > expression > > >> > > sounds > > >> > > > > >>>>>>>>>>>>>>>>>>> good. > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> We > > >> > > > > >>>>>>>>>>>>>>>>> could > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> translate to list of operator names > > internally > > >> > based > > >> > > > on > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> regex. > > >> > > > > >>>>>>>>>>>>>>>>> @Yogi, I went with a list of strings for > > >> attribute > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> because > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> "O2, > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> O3" > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> could > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> be a valid single operator name too :) > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> I am not sure of ways to implement > > anti-affinity > > >> > > across > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> application. > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Though > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> something to consider for later > iteration. > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> Thanks, > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> Isha > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 8:59 PM, Thomas > > >> Weise < > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> [email protected]> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> https://issues.apache.org/jira/browse/SLIDER-82 > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 8:56 PM, Thomas > > Weise > > >> < > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> [email protected]> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> The point was that containers are taken > > away > > >> > from > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> other > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> apps > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> that > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> may > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> have > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> to discard work etc. It's not good style to > > >> claim > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> resources > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> and > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> not > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> use > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> them eventually :-) > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> For this feature it is necessary to look > at > > >> the > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> scheduler > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> capabilities/semantics and > limitations. > > >> For > > >> > > > > example, > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> don't > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> bet > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> exclusively > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> on node requests if the goal is for it to > > work > > >> > with > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> FairScheduler. > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> Also look at Slider, which just recently > > added > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> support > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> for > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> anti-affinity > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> (using node requests). When you run it on > > the > > >> > CDH > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> cluster, > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> it > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> probably > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> won't work... > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 3:19 PM, Pramod > > >> Immaneni < > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> [email protected] > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Once released won't the containers be > > >> > available > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> again > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> in > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> the > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> pool. > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> This > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> would only be optional and not mandatory. > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> Thanks > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> On Tue, Jan 19, 2016 at 2:02 PM, Thomas > > >> Weise < > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> [email protected] > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> How about also supporting a minor > > >> variation > > >> > of > > >> > > it > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> as > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> an > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> option > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> where it greedily gets the total number > > of > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> containers > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> and > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> discards > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ones > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> it > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> can't use and repeats the process for the > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> remaining > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> till > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> everything > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> has > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> been allocated. > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> This is problematic as with resource > > preemption > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> these > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> containers > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> will > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> be > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> potentially taken away from other > > >> applications > > >> > and > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> then > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> thrown > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> away. > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Also does it make sense to support > > >> > anti-cluster > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> affinity? > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> Thanks > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jan 19, 2016 at 1:21 PM, > Isha > > >> > > Arkatkar > > >> > > > < > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> [email protected]> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Hi all, > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> We want add support for > > >> > Anti-affinity > > >> > > in > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Apex > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> allow > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> applications > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> launch specific physical operators on > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> different > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> nodes(APEXCORE-10 > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> < > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > https://issues.apache.org/jira/browse/APEXCORE-10 > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ). > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> Want > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> request > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> your > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> suggestions/ideas for the same! > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> The reasons for using > > anti-affinity > > >> in > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> operators > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> could > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> be: > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ensure > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> reliability, for performance reasons (such > as > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> application > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> may > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> not > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> want > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> 2 > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> i/o intensive operators to land on the > > same > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> node > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> improve > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> performance) > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> or > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> for some application specific > constraints(for > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> example, > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> 2 > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> partitions > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> cannot > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> be run on the same node since they use > > same > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> port > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> number). > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> This > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> is > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> the > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> general rationale for adding Anti-affinity > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> support. > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Since, Yarn does not support > > >> anti-affinity > > >> > yet > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> (YARN-1042 > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> < > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > https://issues.apache.org/jira/browse/YARN-1042 > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> ), > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> we > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> need > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> to > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> implement > > >> > > > > >>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>> the logic in AM. Wanted to get your views > on > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> following > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> aspects > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> for > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> this > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> implementation: > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> *1. How to specify anti-affinity for > > physical > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> operators/partitions > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> in > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> application:* > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> One way for this is to have > an > > >> > > > attribute > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> for > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> setting > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> anti-affinity > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> at > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> the logical operator context. And an > > >> operator > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> can > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> set > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> this > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> attribute > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> with > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> list of operator names which should not be > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> collocated. > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Consider dag with 3 > operators: > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> TestOperator o1 = > > >> > > > > dag.addOperator("O1", > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator()); > > >> > > > > >>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>> TestOperator o2 = > > >> > dag.addOperator("O2", > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator()); > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> TestOperator o3 = > > >> dag.addOperator("O3", > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator()); > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> To set anti-affinity for O1 operator: > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> dag.setAttribute(o1, > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> OperatorContext.ANTI_AFFINITY, > > >> > > > > >>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>> new > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > ArrayList<String>(Arrays.asList("O2", > > >> > > "O3"))); > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> This would mean O1 should > not > > >> be > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> allocated > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> on > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> nodes > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> containing > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> operators O2 and O3. This applies to > all > > >> > > > > >>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>> allocated > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> partitions > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>> of > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> O1, > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> O2, > > >> > > > > >>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>> O3. > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> Also, if same operator name is > > >> part > > >> > of > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> anti-affinity > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> list, > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > > > > >> > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
