Yes, I'm aware of that, but here we have very-long running containers. Nodes are from 100s to 1000s, it depends on the site. Memory and heap sizes are not known.
On Thu, Mar 31, 2016 at 1:56 PM Musty Rehmani <[email protected]> wrote: > Containers are temporary. They shut down after job is finished. How many > nodes in your cluster and what is the memory and heap size > > > > Sent from Yahoo Mail on Android > <https://overview.mail.yahoo.com/mobile/?.src=Android> > > On Thu, Mar 31, 2016 at 4:44 AM, Zoltán Zvara > > <[email protected]> wrote: > The intent to move a container is to improve service-to-service locality, > co-locate services to reduce bottlenecks introduced by network. The idea > that a certain parallel process should be moved comes from an external > service. > > Speculation might not be effective here as I see, since the intent to move > a container can be triggered hourly, daily, weekly or monthly - containers > contain long running services, data pipelines. Or am I on the wrong line > here? Can I simulate something like this with the current speculation? > > Thanks for help! > > On Wed, Mar 30, 2016 at 11:17 PM Eric Payne <[email protected]> > wrote: > >> I think it would help if I knew what the criteria is for wanting to move >> the container. In other words, was the container started on an undesirable >> node in the first place? Or, did the node become undesirable after the >> container started. >> >> Speculation could be considered a "move" operation for containers. If a >> container isn't finishing fast enough, the default speculator will start >> another container on a different node. Would it be possible to create a >> specialized speculator that understood your criteria for needing to move >> containers? If so, it could be done automatically / programatically. >> >> Thanks, >> -Eric >> >> >> ------------------------------ >> *From:* Zoltán Zvara <[email protected]> >> *To:* Vinod Kumar Vavilapalli <[email protected]> >> *Cc:* [email protected] >> *Sent:* Wednesday, March 30, 2016 9:10 AM >> *Subject:* Re: YARN re-locate container >> >> How is this achieved? As far as I see it now, after stopping a container, >> the AM must reallocate the same container with the same resource vector but >> with locality preferences pointed to the new, target node. After the new >> leash has been acquired, then the AM can take it to the new node and >> initiate a `startContainers` message. >> Our use-case with Ericsson would require a more simple API, where (for >> example) a `moveContainer` call from the AM would ask the RM or NM to move >> a container from one node to another (or to any of the specified set of >> preferred nodes). Move would simply kill the container and restart it on >> another node at any given time whenever it is possible - I feel questions >> around scheduling: how container moves should be handled? Probably not like >> simple allocations. >> >> Am I understanding the architecture correctly here? >> >> On Tue, Mar 29, 2016 at 7:31 PM Vinod Kumar Vavilapalli < >> [email protected]> wrote: >> >> Containers can be restarted on other machines already today - YARN just >> leaves it up to the applications to do so. >> >> Are you looking for anything more specifically? >> >> +Vinod >> >> > On Mar 29, 2016, at 9:45 AM, Zoltán Zvara <[email protected]> >> wrote: >> > >> > Dear Hadoop Community, >> > >> > Is there any feature available, or on the road map to support the >> relocation of containers? (Simply restart the container on another machine.) >> > >> > Thanks, >> > Zoltán >> >> >> >>
