Thanks, Amol / that's the solution in a nutshell. I think static ports are fine for our environment so the simpler solution is preferable. I'm hoping to roll out the app tomorrow to give it a shot. Will keep you posted!
________________________________ From: Amol Kekre <[email protected]> Sent: Thursday, March 2, 2017 5:15:42 PM To: [email protected] Subject: Re: Operator Node Affinity Ilya, Put all nodes on the node-balancer list, and only the ones that get the operator-jvm will respond to load-balancer's status url. One place where you have to tweek "do not depend on host/port of distributed OS" is the port number. I believe the load-balancer will use is fixed. You could use a proxy that periodically figures out the port,host and redirects, but then you have an extra hardware hop in between (uptime issue?) that negates the load-balancer play a little. You could do two-proxy servers solution. Thks Amol E:[email protected] | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com<http://www.datatorrent.com> | apex.apache.org *Join us at Apex Big Data World-San Jose <http://www.apexbigdata.com/san-jose.html>, April 4, 2017!* [image: http://www.apexbigdata.com/san-jose-register.html] <http://www.apexbigdata.com/san-jose-register.html> On Thu, Mar 2, 2017 at 8:59 AM, Ganelin, Ilya <[email protected]> wrote: > Thanks – the solution I’m leaning towards is to deploy a load balancer > with a list of the nodes in the cluster, once Apex spins up, the load > balancer should be able to establish connections to the deployed operators > and route data appropriately. > > - Ilya Ganelin > > > On 3/2/17, 8:34 AM, "Amol Kekre" <[email protected]> wrote: > > Ilya, > As Thomas says, attaching a JVM to an operator is do-able, but is > against > the norm in a distributed cluster. A distributed OS cannot guarantee a > node. It could be down or not have resources, .... ZK way or any other > way > to discover post deployment is the way to go. I think a webservice call > through Stram to get the specifics will work too. > > Thks > Amol > > > > E:[email protected] | M: 510-449-2606 | Twitter: @*amolhkekre* > > www.datatorrent.com<http://www.datatorrent.com> | apex.apache.org > > *Join us at Apex Big Data World-San Jose > <http://www.apexbigdata.com/san-jose.html>, April 4, 2017!* > [image: http://www.apexbigdata.com/san-jose-register.html] > <http://www.apexbigdata.com/san-jose-register.html> > > On Wed, Mar 1, 2017 at 8:16 PM, Thomas Weise <[email protected]> wrote: > > > If I understand it correctly you want to run a server in an operator, > > discover its endpoint and push data to it? The preferred way of > doing that > > would be to announce the endpoint through a discovery mechanism > (such as > > ZooKeeper or a shared file) that the upstream entity can use to find > the > > endpoint. > > > > If you are looking for a way to force deploy on a specific node, > then have > > a look at the OperatorContext.LOCALITY_HOST attribute (and also > > AffinityRulesTest). AFAIK you can use a specific host name and the > > scheduler will make best effort to get a container on that host, but > there > > isn't a guarantee. Generally, services running on the cluster > shouldn't > > make assumptions about hosts and ports and use discovery instead. > > > > HTH, > > Thomas > > > > > > On Wed, Mar 1, 2017 at 7:53 PM, Ganelin, Ilya < > [email protected] > > > > > wrote: > > > > > Hello, all – is there any way to deploy a given operator to a > specific > > > Node? E.g. if I’m trying to create a listener for a TCP socket > that can > > > then pipe data to a DAG, is there any way for the location of that > > listener > > > to be deterministic so an upstream entity knows what to connect to? > > > > > > > > > > > > - Ilya Ganelin > > > > > > [image: id:[email protected]] > > > > > > ------------------------------ > > > > > > The information contained in this e-mail is confidential and/or > > > proprietary to Capital One and/or its affiliates and may only be > used > > > solely in performance of work or services for Capital One. The > > information > > > transmitted herewith is intended only for use by the individual or > entity > > > to which it is addressed. If the reader of this message is not the > > intended > > > recipient, you are hereby notified that any review, retransmission, > > > dissemination, distribution, copying or other use of, or taking of > any > > > action in reliance upon this information is strictly prohibited. > If you > > > have received this communication in error, please contact the > sender and > > > delete the material from your computer. > > > > > > > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. > ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
