Generic affinity support in Apex is now official! Thanks Isha for patiently working through a lot of feedback, experimenting with all the different scenarios and even implementing the blacklist based workaround for the CDH scheduler issue.
It would be nice to add to the documentation also :-) I checked and the appropriate place seems to be somewhere around: http://apex.incubator.apache.org/docs/apex/application_development/#streams Isha, it would be great if you can take this up and also include the JSON configuration example. Thanks! On Sun, Feb 28, 2016 at 10:57 PM, Isha Arkatkar <[email protected]> wrote: > Hi Chinmay, > > Please find my replies inline.. > > Thanks, > Isha > On Sun, Feb 28, 2016 at 10:04 PM, Chinmay Kolhatkar <[email protected]> > wrote: > > > Hi Isha, > > > > Couple of points: > > 1. Do the first curly brackets required? They seem to add another level > of > > hierarchy. > > > [ISHA] Yes, for serialization to and from Json, wrapped the list of > rules in a wrapper class. > > 2. operatorRegex, operators, operatorList all seem to mention list list of > > operators. Do you think we can come up with a simpler rather than > providing > > more options? > > > [ISHA] Regex and list are just easier ways to specify operators > affinities instead of pairs. Regex can match any Java regex. I am not too > good with regex, so you saw simple example there :) > > 3. Having JSON innside xml seems a little convoluted to me. I wonder if > > there is a better way to set dt.attr.AFFINITY_RULES_SET. Maybe have t as > > XML itself? > > > [ISHA] I tried XML string codec first too. But we use Hadoop's > Configuration object for properties.xml. And it expects property value to > be a string. So we cannot specify value as XML. Have opened a Jira for > this, but it is actually in Hadoop's code, so we can't have XML string > values unless we override Configuration. > > 4. For operatorRegex which regex language do we support? Is it Java regex > > OR perl/bash based one? > > [ISHA] It is java regex. Simply does a pattern match on all operators in > > dag. It expects at least 2 matches to be found. Otherwise logs warning > and > > does not consider the affinity rule valid. > > > > > > > Thanks, > > Chinmay. > > > > > > > > On Sat, Feb 27, 2016 at 3:38 AM, Isha Arkatkar <[email protected]> > > wrote: > > > > > Hi, > > > > > > To support setting affinity rules from properties.xml, I added json > > > string codec to read the value. I had to wrap the list of rules in an > > > object call AffinityRuleSet to support easy serialization. > > > > > > Here is an example for setting through property: > > > > > > <property> > > > <name>dt.attr.AFFINITY_RULES_SET</name> > > > <value> > > > { > > > "affinityRules": [ > > > { > > > "operatorRegex": "rand|console", > > > "locality": "NODE_LOCAL", > > > "type": "AFFINITY", > > > "relaxLocality": false > > > }, > > > { > > > "operators": { > > > "first": "rand", > > > "second": "passThru" > > > }, > > > "locality": "NODE_LOCAL", > > > "type": "ANTI_AFFINITY", > > > "relaxLocality": false > > > }, > > > { > > > "operatorsList": [ > > > "passThru", > > > "passThru" > > > ], > > > "locality": "NODE_LOCAL", > > > "type": "ANTI_AFFINITY", > > > "relaxLocality": false > > > } > > > ] > > > } > > > </value> > > > </property> > > > > > > Thanks, > > > Isha > > > > > > On Thu, Feb 25, 2016 at 8:16 PM, Chinmay Kolhatkar <[email protected] > > > > > wrote: > > > > > > > Hi Isha, > > > > > > > > 2 ways looks fine. > > > > > > > > Just one point, how would user set affinity rules using DAG content > > > > attribute from properties.xml file? > > > > Can you please provide some example for this? > > > > > > > > Thanks, > > > > Chinmay. > > > > > > > > > > > > On Fri, Feb 26, 2016 at 2:57 AM, Isha Arkatkar <[email protected] > > > > > > wrote: > > > > > > > > > Thanks for the input Chinmay. > > > > > > > > > > The last parameter true/false is for relaxing the constraint. If > > true, > > > > the > > > > > affinity rule would be relaxed if constraint cannot be satisfied. > And > > > if > > > > > false, application will keep requesting and waiting to launch > > > containers > > > > > till all rules are satisfied. > > > > > > > > > > Regarding the dag.setAffinity APIs. I can add those couple APIs in > > > > > LogicalPlan as a wrapper that internally can set the Dag > Attributes. > > > > > Probably yes, that would be easier to use, to set affinity directly > > on > > > > dag > > > > > object. > > > > > > > > > > Then there would 2 ways to set affinity rules: > > > > > > > > > > 1. Through DAG context attribute (Either through code or through > > > > > properties.xml) > > > > > 2. By calling setAffinity on dag. > > > > > > > > > > Thanks, > > > > > Isha > > > > > > > > > > On Wed, Feb 24, 2016 at 10:11 PM, Chinmay Kolhatkar < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > I agree with Pramod. We should probably have single interface > for > > > > > > specifying both if at all possible. This is in interest of not > > > > confusing > > > > > > users with two different configurations at the same time make a > > > > > > locality/affinity API to apear similar to what is out there in > the > > > word > > > > > at > > > > > > the moment. > > > > > > > > > > > > Though it might be early to deprecate stream locality, but after > > some > > > > > time > > > > > > based on user's experience we should consider deprecating Stream > > > > Locality > > > > > > API. > > > > > > > > > > > > @Isha, nice APIs. Questions/suggestions regarding this: > > > > > > 1. What does the "false" signify for AffinityRule constructor. > > > > > > 2. This is a suggestion. Instead of having a List<AffinityRule> > and > > > > then > > > > > > setting this object to attribute of the DAG, should we have an > API > > in > > > > > > LogicalPlan like following: > > > > > > > > > > > > dag.setAffinity(<operator1>, <operator2>, <locality_type>, > > > false/true) > > > > > > > > > > > > dag.setAntiAffinity(<operator1>, <operator2>, false/true) // > Not > > > sure > > > > > > what false/true mean, added for completion. But the point here is > > to > > > > > remove > > > > > > locality for anti-affinity. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Chinmay. > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Feb 25, 2016 at 5:17 AM, Pramod Immaneni < > > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > You could specify it by specifying the names of all operators > > that > > > > are > > > > > > part > > > > > > > of the stream. > > > > > > > > > > > > > > On Wed, Feb 24, 2016 at 3:44 PM, Amol Kekre < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > These are two diff ways of specifying something similar, but > > not > > > > > always > > > > > > > > same. For example, how do I specify the following > > > > > > > > > > > > > > > > This stream has way to high throughput. Force all operators > > that > > > > > > connect > > > > > > > to > > > > > > > > it today (or in future) to be container local. > > > > > > > > > > > > > > > > With stream locality I get that as operator ports are added > to > > > the > > > > > > > stream, > > > > > > > > they automatically inherit stream locality. > > > > > > > > > > > > > > > > Thks, > > > > > > > > Amol > > > > > > > > > > > > > > > > On Wed, Feb 24, 2016 at 3:37 PM, Pramod Immaneni < > > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > To explain the affinity support below seems to be a > super-set > > > > that > > > > > > can > > > > > > > > > consume the stream locality setting whether that was the > > > original > > > > > > > > intention > > > > > > > > > or not. > > > > > > > > > > > > > > > > > > On Wed, Feb 24, 2016 at 3:35 PM, Pramod Immaneni < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Why would we want two different ways of specifying the > same > > > > > thing? > > > > > > > > > > > > > > > > > > > > On Wed, Feb 24, 2016 at 3:29 PM, Amol Kekre < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> Stream locality is a stream attribute. Affinity is > > operator > > > > > > > attribute. > > > > > > > > > >> Technically they may not be connected. Though operator > > > > affinity > > > > > > can > > > > > > > be > > > > > > > > > >> extended to specify a situation that implicity covers > > stream > > > > > > > locality, > > > > > > > > > it > > > > > > > > > >> may be better to let user say the two following > statements > > > > > > > > > independently. > > > > > > > > > >> > > > > > > > > > >> - This stream is heavy on I/O and I want it to be > > ..._local > > > // > > > > > > This > > > > > > > > > >> statement is independent of future additions of > operators > > to > > > > > this > > > > > > > > stream > > > > > > > > > >> - I need these logical operators to not be (or be) on > the > > > same > > > > > > node > > > > > > > > > >> > > > > > > > > > >> Do note that container_local or node_local stream > > localities > > > > > only > > > > > > > > relate > > > > > > > > > >> to > > > > > > > > > >> connected operators. > > > > > > > > > >> > > > > > > > > > >> Thks > > > > > > > > > >> Amol > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> On Wed, Feb 24, 2016 at 1:54 PM, Pramod Immaneni < > > > > > > > > > [email protected]> > > > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > > > >> > Would it make sense to deprecate stream locality in > > favor > > > of > > > > > > this? > > > > > > > > > >> > > > > > > > > > > >> > On Tue, Feb 23, 2016 at 6:05 PM, Isha Arkatkar < > > > > > > > > [email protected]> > > > > > > > > > >> > wrote: > > > > > > > > > >> > > > > > > > > > > >> > > Hi Pramod, > > > > > > > > > >> > > > > > > > > > > > >> > > We can have a list or operators or regex to > specify > > > > > > operators > > > > > > > > > list > > > > > > > > > >> for > > > > > > > > > >> > > affinity rule. Though, internal implementation will > > > > > translate > > > > > > > both > > > > > > > > > >> these > > > > > > > > > >> > > into list of pairs. Since, having pair of operators > > > helps > > > > in > > > > > > > > > >> validation > > > > > > > > > >> > > phase. > > > > > > > > > >> > > > > > > > > > > > >> > > And yes, validation phase will catch conflicting > > > locality > > > > > or > > > > > > > > > affinity > > > > > > > > > >> > > rules and throw validation exception. > > > > > > > > > >> > > > > > > > > > > > >> > > Thanks for input regarding unifiers. It is probably > > > > alright > > > > > > to > > > > > > > > not > > > > > > > > > >> > include > > > > > > > > > >> > > it in the affinity settings. Will leave the current > > > > > > > implementation > > > > > > > > > as > > > > > > > > > >> is. > > > > > > > > > >> > > > > > > > > > > > >> > > Thanks! > > > > > > > > > >> > > Isha > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > On Tue, Feb 23, 2016 at 5:56 PM, Pramod Immaneni < > > > > > > > > > >> [email protected] > > > > > > > > > >> > > > > > > > > > > > >> > > wrote: > > > > > > > > > >> > > > > > > > > > > > >> > > > Isha, > > > > > > > > > >> > > > > > > > > > > > > >> > > > If the implementation would support (eventually) > > > > affinity > > > > > > > > between > > > > > > > > > >> more > > > > > > > > > >> > > than > > > > > > > > > >> > > > 2 logical operators, instead of relying on > > specifying > > > it > > > > > as > > > > > > > > > multiple > > > > > > > > > >> > > rules > > > > > > > > > >> > > > why not make provision in the API to specify more > > > than 2 > > > > > > > logical > > > > > > > > > >> > > operators? > > > > > > > > > >> > > > If there is a stream locality rule that conflicts > > with > > > > the > > > > > > > > > affinity > > > > > > > > > >> > rule > > > > > > > > > >> > > > will the validation catch this? With unifiers the > > > common > > > > > > case > > > > > > > of > > > > > > > > > MxN > > > > > > > > > >> > > > locality is already chosen based on downstream > > > operator. > > > > > In > > > > > > > > other > > > > > > > > > >> cases > > > > > > > > > >> > > > such as single downstream or cascade case I am not > > > sure > > > > it > > > > > > is > > > > > > > > all > > > > > > > > > >> the > > > > > > > > > >> > > > important to have it. > > > > > > > > > >> > > > > > > > > > > > > >> > > > Thanks > > > > > > > > > >> > > > > > > > > > > > > >> > > > On Tue, Feb 23, 2016 at 5:37 PM, Isha Arkatkar < > > > > > > > > > >> [email protected]> > > > > > > > > > >> > > > wrote: > > > > > > > > > >> > > > > > > > > > > > > >> > > > > Hi all, > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > I have opened a review only pull request for > > > > > handling > > > > > > > > > affinity > > > > > > > > > >> > > rules > > > > > > > > > >> > > > in > > > > > > > > > >> > > > > Apex: > > > > > > > > > >> > > > > > > > > > https://github.com/apache/incubator-apex-core/pull/234 > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > Just wanted to confirm with everyone, if the > APIs > > > for > > > > > > > > > specifying > > > > > > > > > >> > > > affinity > > > > > > > > > >> > > > > rules look alright. Also, wanted to get views on > > how > > > > > > > unifiers > > > > > > > > > >> should > > > > > > > > > >> > be > > > > > > > > > >> > > > > handled in case of operator affinity rules. More > > > > details > > > > > > > > follow: > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > I have added a list of affinity rules as an > > > attribute > > > > > in > > > > > > > Dag > > > > > > > > > >> > context. > > > > > > > > > >> > > > For > > > > > > > > > >> > > > > now, have added AffinityRule to be specified for > > > > > operator > > > > > > > > pairs. > > > > > > > > > >> We > > > > > > > > > >> > can > > > > > > > > > >> > > > add > > > > > > > > > >> > > > > regex support in next iteration. > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > Here is sample usage: > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > List<AffinityRule> rules = new ArrayList<>(); > > > > > > > > > >> > > > > // To add node locality between 2 not > > connected > > > > > > > operators > > > > > > > > > rand > > > > > > > > > >> > and > > > > > > > > > >> > > > > console: > > > > > > > > > >> > > > > rules.add(new AffinityRule(Type.AFFINITY, new > > > > > > > OperatorPair( > > > > > > > > > >> > "rand", > > > > > > > > > >> > > > > "console"), Locality.NODE_LOCAL, false)); > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > // To add anti-affinity between all > partitions > > of > > > > > rand > > > > > > > > > operator > > > > > > > > > >> > with > > > > > > > > > >> > > > all > > > > > > > > > >> > > > > partitions of passThru operator: > > > > > > > > > >> > > > > rules.add(new > AffinityRule(Type.ANTI_AFFINITY, > > > new > > > > > > > > > >> OperatorPair( > > > > > > > > > >> > > > "rand", > > > > > > > > > >> > > > > "passThru"), Locality.NODE_LOCAL, false)); > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > // To add anti-affinity between partitions of > > > > > passThru > > > > > > > > > >> operator, > > > > > > > > > >> > > give > > > > > > > > > >> > > > > same operator name in pair: > > > > > > > > > >> > > > > rules.add(new > AffinityRule(Type.ANTI_AFFINITY, > > > new > > > > > > > > > >> OperatorPair( > > > > > > > > > >> > > > > "passThru", "passThru"), Locality.NODE_LOCAL, > > > false)); > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > // Set the rules in dag context > > > > > > > > > >> > > > > dag.setAttribute(DAGContext.AFFINITY_RULES, > > > rules); > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > Please find a sample application using these > > > affinity > > > > > > rules > > > > > > > > > here: > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > https://github.com/ishark/Apex-Samples/tree/master/affinity-example > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > The actual implementation of affinity rules > > heavily > > > > > > depends > > > > > > > > node > > > > > > > > > >> > > specific > > > > > > > > > >> > > > > requests, that was already being handled in > > > > > > > > > >> > StreamingAppMasterService. > > > > > > > > > >> > > To > > > > > > > > > >> > > > > handle node requests for cloudera, I have added > an > > > > > > override > > > > > > > to > > > > > > > > > >> > > blacklist > > > > > > > > > >> > > > > all other nodes except the ones where container > > > > request > > > > > is > > > > > > > to > > > > > > > > be > > > > > > > > > >> > > issued. > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > There is one open question I wanted to bring up: > > In > > > > case > > > > > > of > > > > > > > > > >> > partitioned > > > > > > > > > >> > > > > operators, should the unifiers also follow > > affinity > > > or > > > > > > > > > >> anti-affinity > > > > > > > > > >> > > > rules? > > > > > > > > > >> > > > > Or should they be independent. For now, I have > > kept > > > > them > > > > > > > > > >> independent > > > > > > > > > >> > > and > > > > > > > > > >> > > > > only the actual operators follow affinity rules. > > > > Please > > > > > > > > suggest > > > > > > > > > >> what > > > > > > > > > >> > > > would > > > > > > > > > >> > > > > make more sense from user's specification > > > perspective. > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > Thanks, > > > > > > > > > >> > > > > Isha > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > On Mon, Feb 1, 2016 at 2:12 PM, Isha Arkatkar < > > > > > > > > > >> [email protected]> > > > > > > > > > >> > > > > wrote: > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Hi folks, > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > Summarizing the proposal for > > > > affinity/anti-affinity > > > > > > > rules > > > > > > > > > in > > > > > > > > > >> > apex > > > > > > > > > >> > > as > > > > > > > > > >> > > > > > per discussions in this mail thread. Please > > > suggest > > > > > if I > > > > > > > > > missed > > > > > > > > > >> > > > > something. > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > * For configuration:* > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > - We will have application level > > > > > > affinity/anti-affinity > > > > > > > > > rules. > > > > > > > > > >> > For > > > > > > > > > >> > > > the > > > > > > > > > >> > > > > > first iteration, we can support for > > > (anti-)affinity > > > > > > among > > > > > > > > > >> operators > > > > > > > > > >> > > > > within > > > > > > > > > >> > > > > > single application. We can revisit it in the > > next > > > > > > > iteration > > > > > > > > to > > > > > > > > > >> > > support > > > > > > > > > >> > > > > > across applications. > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > - Each rule will consist of 4 components: > > > > > > > > > >> > > > > > <List of operators>, > > AFFINITY/ANTI-AFFINITY, > > > > > > > > > >> STRICT/RELAXED > > > > > > > > > >> > > > > policy, > > > > > > > > > >> > > > > > CONTAINER/NODE/RACK > > > > > > > > > >> > > > > > I have checked that Apex supports container > > > locality > > > > > > > > between 2 > > > > > > > > > >> > > > operators > > > > > > > > > >> > > > > > that are not connected by Steam. Though did > not > > > find > > > > > an > > > > > > > API > > > > > > > > > for > > > > > > > > > >> > > setting > > > > > > > > > >> > > > > it > > > > > > > > > >> > > > > > through application. So, affinity rules can > > cover > > > > > > > container > > > > > > > > > >> > affinity > > > > > > > > > >> > > as > > > > > > > > > >> > > > > > well. > > > > > > > > > >> > > > > > We will not support THREAD locality in app > level > > > > > > affinity > > > > > > > or > > > > > > > > > >> > > > > > anti-affinity, since it does not apply to > > > > unconnected > > > > > > > > > operators. > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > The list of operators can also be specified > by > > > > regex > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > *For Implementation: * > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > - I have verified that Node specific > requests > > > are > > > > > > > honored > > > > > > > > by > > > > > > > > > >> > > > > HortonWorks > > > > > > > > > >> > > > > > for both Fair and Capacity scheduler. So we > can > > > go > > > > > with > > > > > > > the > > > > > > > > > >> > approach > > > > > > > > > >> > > > of > > > > > > > > > >> > > > > > node specific container requests to implement > > > > > affinity > > > > > > > > rules. > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > - However, we may need to fall back on > > blacklist > > > > > > approach > > > > > > > > for > > > > > > > > > >> CDH > > > > > > > > > >> > > > > distro, > > > > > > > > > >> > > > > > as node specific requests may not work well. > As: > > > > > > > > > >> > > > > > -> With Fair scheduler, node specific > > > > > requests > > > > > > > are > > > > > > > > > not > > > > > > > > > >> > > honored > > > > > > > > > >> > > > > in > > > > > > > > > >> > > > > > CDH 4. Application master keeps sending > requests > > > to > > > > > > Yarn, > > > > > > > > but > > > > > > > > > no > > > > > > > > > >> > > > > containers > > > > > > > > > >> > > > > > are allocated. > > > > > > > > > >> > > > > > -> I could not yet verify the same > with > > > > Fair > > > > > > and > > > > > > > > > >> Capacity > > > > > > > > > >> > > > > > Scheduler on CDH 5. Ran into some > configuration > > > > issues > > > > > > > when > > > > > > > > > >> setting > > > > > > > > > >> > > up > > > > > > > > > >> > > > > CDH > > > > > > > > > >> > > > > > 5.5.1. Will update the thread with findings. > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > I will try to build a prototype for affinity > > > rules > > > > > > with > > > > > > > > > these > > > > > > > > > >> > > things > > > > > > > > > >> > > > in > > > > > > > > > >> > > > > > mind. Please let me know if anyone would like > to > > > > > > > collaborate > > > > > > > > > on > > > > > > > > > >> > this > > > > > > > > > >> > > > > > effort! :) > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > Thanks! > > > > > > > > > >> > > > > > Isha > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > On Mon, Jan 25, 2016 at 10:23 AM, Vlad Rozov < > > > > > > > > > >> > > [email protected]> > > > > > > > > > >> > > > > > wrote: > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > >> Yes, with ability to refer not only to a host > > > name, > > > > > but > > > > > > > to > > > > > > > > > >> > operators > > > > > > > > > >> > > > > >> (possibly both in the same and other > > > applications). > > > > > In > > > > > > > > > general > > > > > > > > > >> > this > > > > > > > > > >> > > > > should > > > > > > > > > >> > > > > >> allow to specify both affinity and anti > > affinity > > > > > using > > > > > > > > > >> > expressions. > > > > > > > > > >> > > > For > > > > > > > > > >> > > > > >> example LOCALITY_CONTAINER = "{app1.op1 or > > > > app1.op2} > > > > > > and > > > > > > > > > >> > > {!app2.op1}". > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> On 1/25/16 09:39, Thomas Weise wrote: > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >>> Agreed. Originally I thought you wanted both > > > > > operators > > > > > > > on > > > > > > > > > the > > > > > > > > > >> > same > > > > > > > > > >> > > > > host. > > > > > > > > > >> > > > > >>> To > > > > > > > > > >> > > > > >>> pin them to the same container the same idea > > can > > > > be > > > > > > > > applied, > > > > > > > > > >> > > through > > > > > > > > > >> > > > a > > > > > > > > > >> > > > > >>> "LOCALITY_CONTAINER" attribute. > > > > > > > > > >> > > > > >>> > > > > > > > > > >> > > > > >>> > > > > > > > > > >> > > > > >>> On Mon, Jan 25, 2016 at 9:18 AM, Vlad Rozov > < > > > > > > > > > >> > > [email protected] > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > >>> wrote: > > > > > > > > > >> > > > > >>> > > > > > > > > > >> > > > > >>> Works as a workaround and requires two extra > > > ports > > > > > > that > > > > > > > > > >> operators > > > > > > > > > >> > > > > >>>> designer > > > > > > > > > >> > > > > >>>> may not necessarily provide out of the box. > > We > > > > may > > > > > > > apply > > > > > > > > > the > > > > > > > > > >> > same > > > > > > > > > >> > > > hack > > > > > > > > > >> > > > > >>>> to > > > > > > > > > >> > > > > >>>> anti-affinity and request that any two > > > operators > > > > > with > > > > > > > > > >> > > anti-affinity > > > > > > > > > >> > > > > rule > > > > > > > > > >> > > > > >>>> are connected by non functional stream, but > > it > > > > does > > > > > > not > > > > > > > > > look > > > > > > > > > >> > like > > > > > > > > > >> > > a > > > > > > > > > >> > > > > good > > > > > > > > > >> > > > > >>>> option :). > > > > > > > > > >> > > > > >>>> > > > > > > > > > >> > > > > >>>> > > > > > > > > > >> > > > > >>>> > > > > > > > > > >> > > > > >>>> On 1/25/16 08:52, Thomas Weise wrote: > > > > > > > > > >> > > > > >>>> > > > > > > > > > >> > > > > >>>> Sandeep suggested to connect them with a > > > stream. > > > > > > > > > >> > > > > >>>>> > > > > > > > > > >> > > > > >>>>> On Mon, Jan 25, 2016 at 8:49 AM, Vlad > Rozov > > < > > > > > > > > > >> > > > [email protected] > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > >>>>> wrote: > > > > > > > > > >> > > > > >>>>> > > > > > > > > > >> > > > > >>>>> Sorry, possibly I miss something. X and Z > > are > > > > not > > > > > > > > directly > > > > > > > > > >> > > > connected > > > > > > > > > >> > > > > >>>>> by a > > > > > > > > > >> > > > > >>>>> > > > > > > > > > >> > > > > >>>>>> stream, how can I apply THREAD_LOCAL? > > > > > > > > > >> > > > > >>>>>> > > > > > > > > > >> > > > > >>>>>> > > > > > > > > > >> > > > > >>>>>> On 1/25/16 08:44, Thomas Weise wrote: > > > > > > > > > >> > > > > >>>>>> > > > > > > > > > >> > > > > >>>>>> Why does THREAD_LOCAL not work? > > > > > > > > > >> > > > > >>>>>> > > > > > > > > > >> > > > > >>>>>>> On Mon, Jan 25, 2016 at 7:43 AM, Vlad > > Rozov > > > < > > > > > > > > > >> > > > > [email protected] > > > > > > > > > >> > > > > >>>>>>> > > > > > > > > > > >> > > > > >>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>> > > > > > > > > > >> > > > > >>>>>>> Neither LOCALITY_HOST or THREAD_LOCAL > > > provides > > > > > > > > required > > > > > > > > > >> > > > > >>>>>>> functionality. > > > > > > > > > >> > > > > >>>>>>> The > > > > > > > > > >> > > > > >>>>>>> > > > > > > > > > >> > > > > >>>>>>> goal is to deploy X and Z to the same > > > > > > > container/JVM. X > > > > > > > > > >> may be > > > > > > > > > >> > > > > >>>>>>>> deployed > > > > > > > > > >> > > > > >>>>>>>> on > > > > > > > > > >> > > > > >>>>>>>> any node. Z needs to be deployed on the > > > same > > > > > > > > container > > > > > > > > > >> as X, > > > > > > > > > >> > > so > > > > > > > > > >> > > > X > > > > > > > > > >> > > > > >>>>>>>> does > > > > > > > > > >> > > > > >>>>>>>> not > > > > > > > > > >> > > > > >>>>>>>> have any host affinity, and Z has > > affinity > > > to > > > > > X, > > > > > > > not > > > > > > > > to > > > > > > > > > >> any > > > > > > > > > >> > > > > specific > > > > > > > > > >> > > > > >>>>>>>> host. > > > > > > > > > >> > > > > >>>>>>>> If database client caches writes, > reading > > > > > inside > > > > > > > the > > > > > > > > > same > > > > > > > > > >> > JVM > > > > > > > > > >> > > > will > > > > > > > > > >> > > > > >>>>>>>> be > > > > > > > > > >> > > > > >>>>>>>> in > > > > > > > > > >> > > > > >>>>>>>> process lookup, while deploying Z to > any > > > > other > > > > > > > > host/JVM > > > > > > > > > >> > leads > > > > > > > > > >> > > to > > > > > > > > > >> > > > > >>>>>>>> inter > > > > > > > > > >> > > > > >>>>>>>> process or inter host lookup. > > > > > > > > > >> > > > > >>>>>>>> > > > > > > > > > >> > > > > >>>>>>>> > > > > > > > > > >> > > > > >>>>>>>> On 1/24/16 21:03, Thomas Weise wrote: > > > > > > > > > >> > > > > >>>>>>>> > > > > > > > > > >> > > > > >>>>>>>> There are the attributes LOCALITY_HOST > > and > > > > > > > > > LOCALITY_RACK > > > > > > > > > >> for > > > > > > > > > >> > > an > > > > > > > > > >> > > > > >>>>>>>> operator > > > > > > > > > >> > > > > >>>>>>>> > > > > > > > > > >> > > > > >>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>> achieve precisely what you are > > describing. > > > > The > > > > > > > value > > > > > > > > > >> would > > > > > > > > > >> > be > > > > > > > > > >> > > > an > > > > > > > > > >> > > > > >>>>>>>>> alias > > > > > > > > > >> > > > > >>>>>>>>> that > > > > > > > > > >> > > > > >>>>>>>>> can be mapped to physical hosts for > > > > > allocation. > > > > > > > The > > > > > > > > > same > > > > > > > > > >> > > could > > > > > > > > > >> > > > be > > > > > > > > > >> > > > > >>>>>>>>> used > > > > > > > > > >> > > > > >>>>>>>>> of > > > > > > > > > >> > > > > >>>>>>>>> anti-affinity, by assigning different > > > > values. > > > > > > > > > >> > > > > >>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>> Stream locality is a special case of > > > > > expressing > > > > > > > > > affinity > > > > > > > > > >> > for > > > > > > > > > >> > > 2 > > > > > > > > > >> > > > > >>>>>>>>> operators. > > > > > > > > > >> > > > > >>>>>>>>> Note that we also need a way to extend > > > this > > > > to > > > > > > > > > >> partitions > > > > > > > > > >> > of > > > > > > > > > >> > > > the > > > > > > > > > >> > > > > >>>>>>>>> same > > > > > > > > > >> > > > > >>>>>>>>> operator. > > > > > > > > > >> > > > > >>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>> On Sun, Jan 24, 2016 at 8:52 PM, Vlad > > > Rozov > > > > < > > > > > > > > > >> > > > > >>>>>>>>> [email protected]> > > > > > > > > > >> > > > > >>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>> It may be quite extreme use case, but > > > > suppose > > > > > > that > > > > > > > > > >> > operator X > > > > > > > > > >> > > > > >>>>>>>>> writes > > > > > > > > > >> > > > > >>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>> a > > > > > > > > > >> > > > > >>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>> distributed in-memory database and > > > operator > > > > Z > > > > > > > reads > > > > > > > > > from > > > > > > > > > >> > it. > > > > > > > > > >> > > Z > > > > > > > > > >> > > > is > > > > > > > > > >> > > > > >>>>>>>>> not > > > > > > > > > >> > > > > >>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>> directly connected to X. In such > cases > > it > > > > may > > > > > > be > > > > > > > > > >> necessary > > > > > > > > > >> > > to > > > > > > > > > >> > > > > let > > > > > > > > > >> > > > > >>>>>>>>>> an > > > > > > > > > >> > > > > >>>>>>>>>> application request container > affinity > > > > > > (deploying > > > > > > > > to > > > > > > > > > >> the > > > > > > > > > >> > > same > > > > > > > > > >> > > > > JVM) > > > > > > > > > >> > > > > >>>>>>>>>> for > > > > > > > > > >> > > > > >>>>>>>>>> operators X and Z as writes may be > > cached > > > > and > > > > > > > reads > > > > > > > > > >> from > > > > > > > > > >> > the > > > > > > > > > >> > > > > same > > > > > > > > > >> > > > > >>>>>>>>>> JVM > > > > > > > > > >> > > > > >>>>>>>>>> will > > > > > > > > > >> > > > > >>>>>>>>>> be potentially faster. > > > > > > > > > >> > > > > >>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>> Technically the same may be applied > to > > > > thread > > > > > > > local > > > > > > > > > on > > > > > > > > > >> > NUMA > > > > > > > > > >> > > > > >>>>>>>>>> boxes, as > > > > > > > > > >> > > > > >>>>>>>>>> NUMA > > > > > > > > > >> > > > > >>>>>>>>>> aware deployment of containers is not > > > > > > supported. > > > > > > > > > >> > > > > >>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>> Vlad > > > > > > > > > >> > > > > >>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>> On 1/23/16 18:01, Yogi Devendra > wrote: > > > > > > > > > >> > > > > >>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>> @Isha > > > > > > > > > >> > > > > >>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>> In my opinion: > > > > > > > > > >> > > > > >>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>> THREAD_LOCAL, CONTAINER_LOCAL on > > stream > > > > is a > > > > > > > > special > > > > > > > > > >> case > > > > > > > > > >> > > of > > > > > > > > > >> > > > > >>>>>>>>>>> generic > > > > > > > > > >> > > > > >>>>>>>>>>> rules > > > > > > > > > >> > > > > >>>>>>>>>>> for Operator X and Operator Y. > > > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>> We can say that, THREAD_LOCAL, > > > > > CONTAINER_LOCAL > > > > > > > > would > > > > > > > > > >> be > > > > > > > > > >> > > > > >>>>>>>>>>> applicable > > > > > > > > > >> > > > > >>>>>>>>>>> only > > > > > > > > > >> > > > > >>>>>>>>>>> if > > > > > > > > > >> > > > > >>>>>>>>>>> operator X and Y are connected by > > > stream. > > > > > But, > > > > > > > way > > > > > > > > > to > > > > > > > > > >> > > express > > > > > > > > > >> > > > > >>>>>>>>>>> this > > > > > > > > > >> > > > > >>>>>>>>>>> should > > > > > > > > > >> > > > > >>>>>>>>>>> be similar to other rules for > > affinity. > > > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>> ~ Yogi > > > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>> On 24 January 2016 at 03:49, Isha > > > > Arkatkar < > > > > > > > > > >> > > > > [email protected] > > > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > > > > >> > > > > >>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>> Hey Chinmay, > > > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>> I certainly agree on common > > set > > > of > > > > > > rules > > > > > > > > for > > > > > > > > > >> > > > > configuring > > > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>> affinity! > > > > > > > > > >> > > > > >>>>>>>>>>>> Well > > > > > > > > > >> > > > > >>>>>>>>>>>> put by a concrete example. :) > > > > > > > > > >> > > > > >>>>>>>>>>>> Only thing I would like to > > > point > > > > > is: > > > > > > > > > >> affinity of > > > > > > > > > >> > > > > >>>>>>>>>>>> operators > > > > > > > > > >> > > > > >>>>>>>>>>>> should > > > > > > > > > >> > > > > >>>>>>>>>>>> not > > > > > > > > > >> > > > > >>>>>>>>>>>> cover thread and container > locality. > > > > Since > > > > > > this > > > > > > > > is > > > > > > > > > >> only > > > > > > > > > >> > > > > >>>>>>>>>>>> applicable > > > > > > > > > >> > > > > >>>>>>>>>>>> when > > > > > > > > > >> > > > > >>>>>>>>>>>> operators are connected by stream. > > So, > > > it > > > > > > makes > > > > > > > > > >> sense to > > > > > > > > > >> > > > have > > > > > > > > > >> > > > > >>>>>>>>>>>> it on > > > > > > > > > >> > > > > >>>>>>>>>>>> Stream > > > > > > > > > >> > > > > >>>>>>>>>>>> rather than in common > configuration. > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> And yes, DAG.validate should > > > only > > > > > > check > > > > > > > > for > > > > > > > > > >> > > REQUIRED > > > > > > > > > >> > > > or > > > > > > > > > >> > > > > >>>>>>>>>>>> STRICT > > > > > > > > > >> > > > > >>>>>>>>>>>> policy. We > > > > > > > > > >> > > > > >>>>>>>>>>>> can agree on one of the > terminologies > > > > > > > > > STRICT/RELAXED > > > > > > > > > >> or > > > > > > > > > >> > > > > >>>>>>>>>>>> REQUIRED/PREFERRED. > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> Thanks! > > > > > > > > > >> > > > > >>>>>>>>>>>> Isha > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> On Fri, Jan 22, 2016 at 8:38 PM, > > > Chinmay > > > > > > > > Kolhatkar > > > > > > > > > < > > > > > > > > > >> > > > > >>>>>>>>>>>> [email protected]> > > > > > > > > > >> > > > > >>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> Hi Isha, Bhupesh, > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> When I suggested singe affinity > > rule, I > > > > was > > > > > > > > mainly > > > > > > > > > >> > talking > > > > > > > > > >> > > > > about > > > > > > > > > >> > > > > >>>>>>>>>>>> "how > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> to" > > > > > > > > > >> > > > > >>>>>>>>>>>>> of configuration and not of > > > > > implementation. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> I see locality is in a way > > suggesting > > > an > > > > > > > > affinity > > > > > > > > > of > > > > > > > > > >> > > > > operators. > > > > > > > > > >> > > > > >>>>>>>>>>>>> They're > > > > > > > > > >> > > > > >>>>>>>>>>>>> close terminologies. > > > > > > > > > >> > > > > >>>>>>>>>>>>> By configuring a locality on > stream, > > > > we're > > > > > > > also, > > > > > > > > > in > > > > > > > > > >> a > > > > > > > > > >> > > way, > > > > > > > > > >> > > > > >>>>>>>>>>>>> defining > > > > > > > > > >> > > > > >>>>>>>>>>>>> affinity of operators. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> Until now, only locality was there > > and > > > > > hence > > > > > > > was > > > > > > > > > >> > straight > > > > > > > > > >> > > > > >>>>>>>>>>>>> forward > > > > > > > > > >> > > > > >>>>>>>>>>>>> in > > > > > > > > > >> > > > > >>>>>>>>>>>>> configuration for user. > > > > > > > > > >> > > > > >>>>>>>>>>>>> Tomorrow, when anti-affinity > > > > configuration > > > > > > > comes > > > > > > > > > up, > > > > > > > > > >> > one > > > > > > > > > >> > > > > might > > > > > > > > > >> > > > > >>>>>>>>>>>>> get > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> confused > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> on how to best use both locality & > > > > > > > > anti-affinity. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> Hence suggested to make both > > > > > > > (locality/affinity & > > > > > > > > > >> > > > > >>>>>>>>>>>> anti-affinity) as > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> a > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> part > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> of single configuration. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> Suggestion is to have a more > commonly > > > > > adopted > > > > > > > > > >> > > configuration > > > > > > > > > >> > > > > >>>>>>>>>>>> which > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> admins > > > > > > > > > >> > > > > >>>>>>>>>>>>> and developer's are familiar with. > > > > > > > > > >> > > > > >>>>>>>>>>>>> Again referring to vShere > > Hypervisor's > > > > > > > affinity > > > > > > > > > >> rules. > > > > > > > > > >> > I > > > > > > > > > >> > > > > think > > > > > > > > > >> > > > > >>>>>>>>>>>>> they > > > > > > > > > >> > > > > >>>>>>>>>>>>> have > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> a > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> single configuration which does > > both. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> Having said that, here is a quick > > > > > suggestion > > > > > > on > > > > > > > > how > > > > > > > > > >> both > > > > > > > > > >> > > can > > > > > > > > > >> > > > > be > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> achieved > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> in > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> a single configuration: > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> CATEGORY TYPE POLICY > > > > > > > > > >> ENTITIES > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> Affinity THREAD > > > > REQUIRED > > > > > > > > > >> O1, O2 > > > > > > > > > >> > > > > >>>>>>>>>>>>> //Meaning > > > > > > > > > >> > > > > >>>>>>>>>>>>> Operator1 & Operator2 should be > > thread > > > > > local > > > > > > > > > >> > > > > >>>>>>>>>>>>> Affinity NODE > > > > > PREFERRED > > > > > > > > > >> O3, O4 > > > > > > > > > >> > > > > >>>>>>>>>>>>> //Meaning > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> O3 & > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> O4 are preferred to be in node > local > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> AntiAffinity NODE > > > REQUIRED > > > > > > > > > O1, > > > > > > > > > >> O4 > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> //Meaning > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> O1 & > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> O4 should not be on same node. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> AntiAffinity RACK > > > > PREFERRED > > > > > > > > O2, > > > > > > > > > O4 > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> //Meaning > > > > > > > > > >> > > > > >>>>>>>>>>>>> O2 > > > > > > > > > >> > > > > >>>>>>>>>>>>> & > > > > > > > > > >> > > > > >>>>>>>>>>>>> O4 are preferred not to be on same > > > rack. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> Linux setting affinity of CPUs for > > > > threads > > > > > > is > > > > > > > > > >> another > > > > > > > > > >> > way > > > > > > > > > >> > > > of > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> configuration > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> we can take a look at. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> Learning from these commonly > adopted > > > > > > > > configuration > > > > > > > > > >> > > pattern, > > > > > > > > > >> > > > we > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> should > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> come > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> up with best configuration > suitable > > > for > > > > > > > > > distributed > > > > > > > > > >> > > > > >>>>>>>>>>>>> environment. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> Idea here is to not have our own > > > > > > configuration > > > > > > > > and > > > > > > > > > >> give > > > > > > > > > >> > > > > >>>>>>>>>>>> something > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> new > > > > > > > > > >> > > > > >>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>> the users. Otherwise such an > > important > > > > > > concept > > > > > > > > > might > > > > > > > > > >> > > > quickly > > > > > > > > > >> > > > > >>>>>>>>>>>>> get > > > > > > > > > >> > > > > >>>>>>>>>>>>> lost. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> Regarding the DAG.validate I think > > we > > > > > would > > > > > > > need > > > > > > > > > to > > > > > > > > > >> add > > > > > > > > > >> > > > some > > > > > > > > > >> > > > > >>>>>>>>>>>>> new > > > > > > > > > >> > > > > >>>>>>>>>>>>> stuff > > > > > > > > > >> > > > > >>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>> take care of anti-affinity. > > > > > > > > > >> > > > > >>>>>>>>>>>>> Plus, anti-affinity/affinity > should > > be > > > > > > > validated > > > > > > > > > at > > > > > > > > > >> > > > > >>>>>>>>>>>>> DAG.validate > > > > > > > > > >> > > > > >>>>>>>>>>>>> only > > > > > > > > > >> > > > > >>>>>>>>>>>>> for > > > > > > > > > >> > > > > >>>>>>>>>>>>> the ones which are required. > > > > > > > > > >> > > > > >>>>>>>>>>>>> For preferred policies, validation > > in > > > > > > logical > > > > > > > > plan > > > > > > > > > >> > might > > > > > > > > > >> > > > be a > > > > > > > > > >> > > > > >>>>>>>>>>>>> early > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> check. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> Thanks, > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>> Chinmay. > > > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> On Sat, Jan 23, 2016 at 3:15 AM, > > Isha > > > > > > > Arkatkar < > > > > > > > > > >> > > > > >>>>>>>>>>>>> [email protected]> > > > > > > > > > >> > > > > >>>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> Hi, > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> Thanks for inputs! I like > the > > > > idea > > > > > of > > > > > > > > > having > > > > > > > > > >> > > single > > > > > > > > > >> > > > > set > > > > > > > > > >> > > > > >>>>>>>>>>>>> of > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> (anti?)affinity rules at dag > context > > > > > level. > > > > > > > > > >> > > > > >>>>>>>>>>>>>> There could still be > > conflicts > > > > > based > > > > > > > on > > > > > > > > > >> > Locality > > > > > > > > > >> > > > for > > > > > > > > > >> > > > > >>>>>>>>>>>>>> Streams > > > > > > > > > >> > > > > >>>>>>>>>>>>>> or > > > > > > > > > >> > > > > >>>>>>>>>>>>>> Node > > > > > > > > > >> > > > > >>>>>>>>>>>>>> locality attribute set for > > operators. > > > > But > > > > > > as > > > > > > > > > >> Sandeep > > > > > > > > > >> > > > > >>>>>>>>>>>>>> suggested, I > > > > > > > > > >> > > > > >>>>>>>>>>>>>> also > > > > > > > > > >> > > > > >>>>>>>>>>>>>> think Dag.validate should fail in > > > case > > > > of > > > > > > > > > >> > contradicting > > > > > > > > > >> > > > > >>>>>>>>>>>>>> constraints. > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> We currently do not have > > > > 'affinity' > > > > > > > among > > > > > > > > > non > > > > > > > > > >> > > stream > > > > > > > > > >> > > > > >>>>>>>>>>>>>> operators, as > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> Bhupesh > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> pointed out. It is somewhat > > > achievable > > > > by > > > > > > > > > >> requesting > > > > > > > > > >> > > node > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> locality > > > > > > > > > >> > > > > >>>>>>>>>>>>> for > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> operators (Assuming node requests > > work > > > > as > > > > > > > > > expected). > > > > > > > > > >> > But > > > > > > > > > >> > > > > >>>>>>>>>>>>> should we > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> consider > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> adding affinity specifications > > > support > > > > as > > > > > > > well > > > > > > > > > >> along > > > > > > > > > >> > > with > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> anti-affinity? > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> Regarding specification of > > > > > attributes > > > > > > > > from > > > > > > > > > >> > > > > dt-site.xml. > > > > > > > > > >> > > > > >>>>>>>>>>>>>> We > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> can > > > > > > > > > >> > > > > >>>>>>>>>>>>> go > > > > > > > > > >> > > > > >>>>>>>>>>>>> with > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> Json like string or even xml > > > > > representation > > > > > > > for > > > > > > > > > >> complex > > > > > > > > > >> > > > > >>>>>>>>>>>>> objects. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> What > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> is > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> our current behavior for setting > > Java > > > > > > object > > > > > > > > > >> > properties > > > > > > > > > >> > > > > >>>>>>>>>>>>>> through > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> XML? > > > > > > > > > >> > > > > >>>>>>>>>>>>> We > > > > > > > > > >> > > > > >>>>>>>>>>>>> can > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> follow the same for this as well. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> As for precedence or > ability > > to > > > > > > satisfy > > > > > > > > > >> > > constraints: > > > > > > > > > >> > > > > >>>>>>>>>>>>> Right > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> now in > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> normal > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> scenario, if resources are not > > > > available > > > > > > for > > > > > > > > > >> > allocating > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> containers, > > > > > > > > > >> > > > > >>>>>>>>>>>>> we > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> keep > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> sending to request till all are > > > > obtained. > > > > > > > > > >> Likewise, in > > > > > > > > > >> > > > case > > > > > > > > > >> > > > > of > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> strict > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> anti-affinity policy, we should > keep > > > the > > > > > > > > > >> application in > > > > > > > > > >> > > > > >>>>>>>>>>>>> ACCEPTED > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> state > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> till > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> the anti-affinity constraint is > > > > > satisfied. > > > > > > > For > > > > > > > > > >> relaxed > > > > > > > > > >> > > > > >>>>>>>>>>>>>> policy, we > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> can > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> decide timeout for relaxing the > > > > > > anti-affinity > > > > > > > > > rule. > > > > > > > > > >> > > Please > > > > > > > > > >> > > > > note > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> this > > > > > > > > > >> > > > > >>>>>>>>>>>>>> applies only when we have > > > > > non-contradicting > > > > > > > > > rules. > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> Thanks! > > > > > > > > > >> > > > > >>>>>>>>>>>>>> Isha > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 5:30 AM, > > > > Bhupesh > > > > > > > > Chawda < > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> [email protected] > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> I agree on having a single set of > > > rules > > > > > for > > > > > > > the > > > > > > > > > >> > affinity > > > > > > > > > >> > > as > > > > > > > > > >> > > > > >>>>>>>>>>>>> well > > > > > > > > > >> > > > > >>>>>>>>>>>>> as > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> anti-affinity of operators / > > > partitions > > > > on > > > > > > > > > >> containers. > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> However, I noted the following > > > points: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> 1. AFAIK, we do not > > support > > > > > > affinity > > > > > > > > > >> > (locality) > > > > > > > > > >> > > > in > > > > > > > > > >> > > > > a > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> general > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> sense. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> The > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> affinity is only for a > > > stream, > > > > > not > > > > > > > for > > > > > > > > > >> *any* > > > > > > > > > >> > two > > > > > > > > > >> > > > > >>>>>>>>>>>>> operators. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> So, > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> we > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> should also look at the > > > > general > > > > > > case > > > > > > > > and > > > > > > > > > >> see > > > > > > > > > >> > > how > > > > > > > > > >> > > > it > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> can > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> be > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> supported, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> if > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> there are valid use > cases. > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> 2. Coming to > anti-affinity, > > > we > > > > > > cannot > > > > > > > > > type > > > > > > > > > >> > cast > > > > > > > > > >> > > it > > > > > > > > > >> > > > > as > > > > > > > > > >> > > > > >>>>>>>>>>>>>> a > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> type of > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> affinity > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> rule. Saying "two > > operators > > > > must > > > > > > be > > > > > > > on > > > > > > > > > the > > > > > > > > > >> > same > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> container" is > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> very > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> different from saying > > "these > > > > two > > > > > > > > > operators > > > > > > > > > >> > must > > > > > > > > > >> > > > not > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> be on > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> the > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> same > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> container". In this > case, > > > the > > > > > > second > > > > > > > > one > > > > > > > > > >> is a > > > > > > > > > >> > > > much > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> more > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> relaxed > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> rule > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> as > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> compared to the first > one. > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> 3. Once we have this > generic > > > set > > > > > of > > > > > > > > rules, > > > > > > > > > >> > there > > > > > > > > > >> > > > must > > > > > > > > > >> > > > > >>>>>>>>>>>>>> be a > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> "satisfiability" test > run > > > > before > > > > > > > > > >> requesting > > > > > > > > > >> > > YARN > > > > > > > > > >> > > > > for > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> containers. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> If > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> the > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> request is not > satisfiable, > > > > then > > > > > > > there > > > > > > > > is > > > > > > > > > >> no > > > > > > > > > >> > > point > > > > > > > > > >> > > > > >>>>>>>>>>>>> asking > > > > > > > > > >> > > > > >>>>>>>>>>>>> YARN > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> allocate > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> containers in this > manner. > > > In > > > > > case > > > > > > > it > > > > > > > > is > > > > > > > > > >> not > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> satisfiable, we > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> must > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> also > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> have > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> a default order in which > > the > > > > > rules > > > > > > > can > > > > > > > > be > > > > > > > > > >> > > > "relaxed" > > > > > > > > > >> > > > > >>>>>>>>>>>>>> and > > > > > > > > > >> > > > > >>>>>>>>>>>>>> the > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> request > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> be > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> made > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> satisfiable. For example, > > some > > > > > very > > > > > > > > strict > > > > > > > > > >> > rules > > > > > > > > > >> > > > may > > > > > > > > > >> > > > > be > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> ignored, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> or > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> made > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> less constraining ( for > > > example > > > > > "on > > > > > > > the > > > > > > > > > >> same > > > > > > > > > >> > > > > >>>>>>>>>>>>> container" => > > > > > > > > > >> > > > > >>>>>>>>>>>>> "on > > > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>> the > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> same > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> node"). > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> Thanks. > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> -Bhupesh > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:54 PM, > > > > > Aniruddha > > > > > > > > > >> Thombare < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> [email protected]> > wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> +1 On Chinmay's suggestion about > > > > having > > > > > > > single > > > > > > > > > >> set of > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> affinity > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> rules. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> Thanks, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> Aniruddha > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:57 > PM, > > > > > Sandeep > > > > > > > > > >> Deshmukh < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> [email protected] > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> Between 2 operators, if one > > > configures > > > > > > > > > >> > thread/container > > > > > > > > > >> > > > > local > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> and > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> anti-affinity as well, which > one > > > will > > > > > > take > > > > > > > > > >> affect? > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> The DAG validation step should > > > error > > > > > out > > > > > > > in > > > > > > > > > this > > > > > > > > > >> > > case. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> +1 on suggestion by Chinmay to > > name > > > > it > > > > > > > > > "Affinity > > > > > > > > > >> > > Rules" > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> than > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> anti-affinity. We are just > > extending > > > > our > > > > > > > > > container > > > > > > > > > >> > > > > allocation > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> scheme > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> support containers not to be > > > > allocated > > > > > > > > > together. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> Regards, > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> Sandeep > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:43 > PM, > > > > > Chinmay > > > > > > > > > >> Kolhatkar > > > > > > > > > >> > < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> [email protected]> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> Hi Isha, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> Couple of points: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> 1. About the interface to > > > > configuring > > > > > > > > > >> > anti-affinity, > > > > > > > > > >> > > as > > > > > > > > > >> > > > > per > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> suggestion > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> above there are 2 different > way > > > to > > > > > > > > configure > > > > > > > > > >> > > locality > > > > > > > > > >> > > > > and > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> anti-affinity: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> i.e. dag.setAttribute - for > > > > > anti-affinity > > > > > > > & > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > dag.addStream(...).setLocality > > > > > for > > > > > > > > > >> locality. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> Between 2 operators, if one > > > > configures > > > > > > > > > >> > > thread/container > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> local > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> and > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> anti-affinity as well, which > > one > > > > will > > > > > > > take > > > > > > > > > >> affect? > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> 2. Consider there could be > such > > > > > > confusion > > > > > > > as > > > > > > > > > >> above, > > > > > > > > > >> > > > would > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> it > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> make > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> sense > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> have a single API which takes > care > > > of > > > > > both > > > > > > > > > >> > > anti-affinity > > > > > > > > > >> > > > > and > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> locality. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> This > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> way, one is configurable. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> 3. This point is coming from > how > > VM > > > > > > > affinity > > > > > > > > is > > > > > > > > > >> > > > configured > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> in > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> vSphere. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> The VMs are configured > affinity > > > are > > > > > > > called > > > > > > > > as > > > > > > > > > >> > > > "affinity > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> rules" > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> and > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> not > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> "anti-affinity rules". > > Ultimately > > > > idea > > > > > > is > > > > > > > to > > > > > > > > > >> > allocate > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> processing > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> nodes. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> Via "VM-VM affinity rules", > > > > > anti-affinity > > > > > > is > > > > > > > > > also > > > > > > > > > >> > > > > configured. > > > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>> But > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> there > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> is > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> a single set of rule definition > > for > > > > both > > > > > > > > > affinity > > > > > > > > > >> > > > (similar > > > > > > > > > >> > > > > to > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> locality > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> in > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> our case) and anti-affinity. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> Would it be a better approach > for > > > > > > > configuring > > > > > > > > > >> > locality > > > > > > > > > >> > > > > rules > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> and > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> anti-affinity rules in a > single > > > > rule > > > > > > and > > > > > > > > call > > > > > > > > > >> it > > > > > > > > > >> > > > > "affinity > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> rule". > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> Thanks, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> Chinmay. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 12:24 > PM, > > > Yogi > > > > > > > > Devendra > > > > > > > > > < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> [email protected] > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> @Isha > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> I understand that > anti-affinity > > > > across > > > > > > > > > >> application > > > > > > > > > >> > is > > > > > > > > > >> > > > not > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> straight-forward. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> It would be OK even if we do > > not > > > > > have > > > > > > it > > > > > > > > in > > > > > > > > > >> > > iteration > > > > > > > > > >> > > > > 1. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> But, for attributes syntax; I > > > still > > > > > > think > > > > > > > > > that > > > > > > > > > >> > Java > > > > > > > > > >> > > > > object > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> should > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> be > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> avoided as they will be hard > to > > > > > > configure > > > > > > > > > from > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> dt-site.xml or > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> other > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> config > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> files. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> Other suggestion for this could > > be > > > > JSON > > > > > > > > > >> > representation > > > > > > > > > >> > > > of > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> String > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> array: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> ["O2", "O3"]. (If operator > > names > > > > has > > > > > > > some > > > > > > > > > >> special > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> characters > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> like > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> " > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> or [ > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> or , those will be escaped in > the > > > > JSON > > > > > > > > > >> > > representation.) > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> Not sure if others agree on > this; > > > but > > > > > > > > attribute > > > > > > > > > >> > syntax > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> should > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> be > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> finalized > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> in iteration 1 to avoid > > backward > > > > > > > > > compatibility > > > > > > > > > >> > > issues > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> later. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> ~ Yogi > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> On 22 January 2016 at 00:43, > > Thomas > > > > > > Weise < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> [email protected]> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> Node based requests is the > best > > > > > > approach > > > > > > > - > > > > > > > > if > > > > > > > > > >> it > > > > > > > > > >> > > works > > > > > > > > > >> > > > > :-) > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> Blacklisting will require to > > > allocate > > > > > the > > > > > > > > > >> containers > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> sequentially. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> It > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> will > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> work, but slow down > application > > > > > startup, > > > > > > > > > >> especially > > > > > > > > > >> > > for > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> larger > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> topologies. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jan 21, 2016 at > 10:42 > > > AM, > > > > > Isha > > > > > > > > > >> Arkatkar < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> [email protected]> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> Hi, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> Should we consider the > > > node > > > > > > based > > > > > > > > > >> requests > > > > > > > > > >> > if > > > > > > > > > >> > > > it > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> works > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> with > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Capacity > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> Scheduler or avoid 2b > > approach > > > > > > > > altogether? > > > > > > > > > I > > > > > > > > > >> > > checked > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> that > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> node > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> requests > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> do > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> not work with fair scheduler > on > > > CDH > > > > > > > > cluster. > > > > > > > > > >> Yarn > > > > > > > > > >> > > does > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> not > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> return > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> any > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> container if hostname is > given > > in > > > > the > > > > > > > > > container > > > > > > > > > >> > > > request. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> I > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> am > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> trying > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> setup a small virtual > hortonworks > > > > > cluster > > > > > > > to > > > > > > > > > >> check > > > > > > > > > >> > the > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> this > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> behavior > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> on > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> that. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> YARN-2027 < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > https://issues.apache.org/jira/browse/YARN-2027 > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> mentioned > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> that > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> container requests are not > > > honored > > > > in > > > > > > > > > capacity > > > > > > > > > >> > > > scheduler > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> too. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> But I > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> am > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> not > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> sure if it is because of > distro > > > > > > dependent > > > > > > > > > issue. > > > > > > > > > >> > > Please > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> share > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> insights. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> @Vlad, Adding support for > > > regular > > > > > > > > > expression > > > > > > > > > >> > > sounds > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> good. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> We > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> could > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> translate to list of operator > > > names > > > > > > > > > internally > > > > > > > > > >> > based > > > > > > > > > >> > > > on > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> regex. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> @Yogi, I went with a list of > > > > strings > > > > > > for > > > > > > > > > >> attribute > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> because > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> "O2, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> O3" > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> could > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> be a valid single operator name > > too > > > > :) > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> I am not sure of ways to > > implement > > > > > > > > > anti-affinity > > > > > > > > > >> > > across > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> application. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Though > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> something to consider for > > later > > > > > > > > iteration. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> Thanks, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> Isha > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at > 8:59 > > > PM, > > > > > > > Thomas > > > > > > > > > >> Weise < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> [email protected]> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> https://issues.apache.org/jira/browse/SLIDER-82 > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 8:56 > > PM, > > > > > > Thomas > > > > > > > > > Weise > > > > > > > > > >> < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> [email protected]> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> The point was that > > containers > > > > are > > > > > > > taken > > > > > > > > > away > > > > > > > > > >> > from > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> other > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> apps > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> that > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> may > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> have > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> to discard work etc. It's not > > good > > > > > style > > > > > > > to > > > > > > > > > >> claim > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> resources > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> and > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> not > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> use > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> them eventually :-) > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> For this feature it is > > necessary > > > > to > > > > > > look > > > > > > > > at > > > > > > > > > >> the > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> scheduler > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> capabilities/semantics > and > > > > > > > > limitations. > > > > > > > > > >> For > > > > > > > > > >> > > > > example, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> don't > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> bet > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> exclusively > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> on node requests if the goal > is > > > for > > > > > it > > > > > > to > > > > > > > > > work > > > > > > > > > >> > with > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> FairScheduler. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> Also look at Slider, which > > just > > > > > > recently > > > > > > > > > added > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> support > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> for > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> anti-affinity > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> (using node requests). When > > you > > > > run > > > > > > it > > > > > > > on > > > > > > > > > the > > > > > > > > > >> > CDH > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> cluster, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> it > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> probably > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> won't work... > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 3:19 > > PM, > > > > > Pramod > > > > > > > > > >> Immaneni < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> [email protected] > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Once released won't the > > > > > containers > > > > > > be > > > > > > > > > >> > available > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> again > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> in > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> the > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> pool. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> This > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> would only be optional and not > > > > > mandatory. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> Thanks > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> On Tue, Jan 19, 2016 at > 2:02 > > > PM, > > > > > > Thomas > > > > > > > > > >> Weise < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> [email protected] > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> How about also > supporting > > a > > > > > minor > > > > > > > > > >> variation > > > > > > > > > >> > of > > > > > > > > > >> > > it > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> as > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> an > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> option > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> where it greedily gets > the > > > > total > > > > > > > number > > > > > > > > > of > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> containers > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> and > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> discards > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ones > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> it > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> can't use and repeats the > > > process > > > > > for > > > > > > > the > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> remaining > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> till > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> everything > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> has > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> been allocated. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> This is problematic as with > > > > resource > > > > > > > > > preemption > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> these > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> containers > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> will > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> be > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> potentially taken away from > > > other > > > > > > > > > >> applications > > > > > > > > > >> > and > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> then > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> thrown > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> away. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Also does it make sense > to > > > > > support > > > > > > > > > >> > anti-cluster > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> affinity? > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> Thanks > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jan 19, 2016 > at > > > 1:21 > > > > > PM, > > > > > > > > Isha > > > > > > > > > >> > > Arkatkar > > > > > > > > > >> > > > < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> [email protected]> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Hi all, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> We want add > > support > > > > for > > > > > > > > > >> > Anti-affinity > > > > > > > > > >> > > in > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Apex > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> allow > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> applications > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> launch specific physical > > operators > > > > on > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> different > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> nodes(APEXCORE-10 > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > https://issues.apache.org/jira/browse/APEXCORE-10 > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ). > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> Want > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> request > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> your > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> suggestions/ideas for the > > same! > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> The reasons for > > using > > > > > > > > > anti-affinity > > > > > > > > > >> in > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> operators > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> could > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> be: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ensure > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> reliability, for performance > > > > reasons > > > > > > > (such > > > > > > > > as > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> application > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> may > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> not > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> want > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> 2 > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> i/o intensive operators > to > > > land > > > > > on > > > > > > > the > > > > > > > > > same > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> node > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> improve > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> performance) > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> or > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> for some application specific > > > > > > > > constraints(for > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> example, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> 2 > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> partitions > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> cannot > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> be run on the same node > > since > > > > > they > > > > > > > use > > > > > > > > > same > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> port > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> number). > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> This > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> is > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> the > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> general rationale for adding > > > > > > Anti-affinity > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> support. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Since, Yarn does not > > > support > > > > > > > > > >> anti-affinity > > > > > > > > > >> > yet > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> (YARN-1042 > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> < > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > https://issues.apache.org/jira/browse/YARN-1042 > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> ), > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> we > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> need > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> to > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> implement > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> the logic in AM. Wanted to > get > > > > your > > > > > > > views > > > > > > > > on > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> following > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> aspects > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> for > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> this > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> implementation: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> *1. How to specify > > > anti-affinity > > > > > for > > > > > > > > > physical > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> operators/partitions > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> in > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> application:* > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> One way for > this > > > is > > > > to > > > > > > > have > > > > > > > > an > > > > > > > > > >> > > > attribute > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> for > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> setting > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> anti-affinity > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> at > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> the logical operator > > context. > > > > And > > > > > > an > > > > > > > > > >> operator > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> can > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> set > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> this > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> attribute > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> with > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> list of operator names which > > > should > > > > > not > > > > > > > be > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> collocated. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Consider dag > > with 3 > > > > > > > > operators: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > TestOperator > > > o1 = > > > > > > > > > >> > > > > dag.addOperator("O1", > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator()); > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> TestOperator o2 = > > > > > > > > > >> > dag.addOperator("O2", > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator()); > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> TestOperator o3 = > > > > > > > > > >> dag.addOperator("O3", > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator()); > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> To set anti-affinity for > > O1 > > > > > > > operator: > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > dag.setAttribute(o1, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > OperatorContext.ANTI_AFFINITY, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> new > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > ArrayList<String>(Arrays.asList("O2", > > > > > > > > > >> > > "O3"))); > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> This would > mean > > > O1 > > > > > > should > > > > > > > > not > > > > > > > > > >> be > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> allocated > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> on > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> nodes > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> containing > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> operators O2 and O3. This > > > > applies > > > > > > to > > > > > > > > all > > > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>> allocated > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> partitions > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> of > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> O1, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> O2, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> O3. > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> Also, if same > > > operator > > > > > > name > > > > > > > is > > > > > > > > > >> part > > > > > > > > > >> > of > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> anti-affinity > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> list, > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
