Ilya, Put all nodes on the node-balancer list, and only the ones that get the operator-jvm will respond to load-balancer's status url. One place where you have to tweek "do not depend on host/port of distributed OS" is the port number. I believe the load-balancer will use is fixed. You could use a proxy that periodically figures out the port,host and redirects, but then you have an extra hardware hop in between (uptime issue?) that negates the load-balancer play a little. You could do two-proxy servers solution.
Thks Amol E:[email protected] | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com | apex.apache.org *Join us at Apex Big Data World-San Jose <http://www.apexbigdata.com/san-jose.html>, April 4, 2017!* [image: http://www.apexbigdata.com/san-jose-register.html] <http://www.apexbigdata.com/san-jose-register.html> On Thu, Mar 2, 2017 at 8:59 AM, Ganelin, Ilya <[email protected]> wrote: > Thanks – the solution I’m leaning towards is to deploy a load balancer > with a list of the nodes in the cluster, once Apex spins up, the load > balancer should be able to establish connections to the deployed operators > and route data appropriately. > > - Ilya Ganelin > > > On 3/2/17, 8:34 AM, "Amol Kekre" <[email protected]> wrote: > > Ilya, > As Thomas says, attaching a JVM to an operator is do-able, but is > against > the norm in a distributed cluster. A distributed OS cannot guarantee a > node. It could be down or not have resources, .... ZK way or any other > way > to discover post deployment is the way to go. I think a webservice call > through Stram to get the specifics will work too. > > Thks > Amol > > > > E:[email protected] | M: 510-449-2606 | Twitter: @*amolhkekre* > > www.datatorrent.com | apex.apache.org > > *Join us at Apex Big Data World-San Jose > <http://www.apexbigdata.com/san-jose.html>, April 4, 2017!* > [image: http://www.apexbigdata.com/san-jose-register.html] > <http://www.apexbigdata.com/san-jose-register.html> > > On Wed, Mar 1, 2017 at 8:16 PM, Thomas Weise <[email protected]> wrote: > > > If I understand it correctly you want to run a server in an operator, > > discover its endpoint and push data to it? The preferred way of > doing that > > would be to announce the endpoint through a discovery mechanism > (such as > > ZooKeeper or a shared file) that the upstream entity can use to find > the > > endpoint. > > > > If you are looking for a way to force deploy on a specific node, > then have > > a look at the OperatorContext.LOCALITY_HOST attribute (and also > > AffinityRulesTest). AFAIK you can use a specific host name and the > > scheduler will make best effort to get a container on that host, but > there > > isn't a guarantee. Generally, services running on the cluster > shouldn't > > make assumptions about hosts and ports and use discovery instead. > > > > HTH, > > Thomas > > > > > > On Wed, Mar 1, 2017 at 7:53 PM, Ganelin, Ilya < > [email protected] > > > > > wrote: > > > > > Hello, all – is there any way to deploy a given operator to a > specific > > > Node? E.g. if I’m trying to create a listener for a TCP socket > that can > > > then pipe data to a DAG, is there any way for the location of that > > listener > > > to be deterministic so an upstream entity knows what to connect to? > > > > > > > > > > > > - Ilya Ganelin > > > > > > [image: id:[email protected]] > > > > > > ------------------------------ > > > > > > The information contained in this e-mail is confidential and/or > > > proprietary to Capital One and/or its affiliates and may only be > used > > > solely in performance of work or services for Capital One. The > > information > > > transmitted herewith is intended only for use by the individual or > entity > > > to which it is addressed. If the reader of this message is not the > > intended > > > recipient, you are hereby notified that any review, retransmission, > > > dissemination, distribution, copying or other use of, or taking of > any > > > action in reliance upon this information is strictly prohibited. > If you > > > have received this communication in error, please contact the > sender and > > > delete the material from your computer. > > > > > > > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. >
