The REST API doesn't work; we need YARN to proxy the requests (YARN-2031). The IPC API does support flexing -the caller submits a new resources.json file which is received by the AM, saved to HDFS (so that on a restart or stop/start the values preserved), then acted on. The AM reviews the #of containers and adds/removes them.
We don't explicitly have a way for the running application to change the values. this is actually something the storm team would like. Again, the IPC channel is there, it's just currently set up on the expectation that the user is tuning the cluster size. One thing we have avoided doing is any attempt to dynamically flex cluster size based on load, the way Apache Helix does. It's hard to decide when to scale up or down, and in a multi-component system, the scaling is often across the entire system, rather than just one layer. We've concluded that (currently) it's best to publish load information and let other tools make those decisions. -steve > On 25 Jul 2015, at 07:12, Yong Feng <[email protected]> wrote: > > Regarding to Slider API, please refer to > https://github.com/apache/incubator-slider/tree/develop/slider-core/src/main/java/org/apache/slider/client > . > > Ambari has Slider View to deploy and manage Slider-based application. You > could refer to source codes of Ambari Slider View as an example how to call > Slider API as follows > > https://github.com/apache/ambari/blob/trunk/contrib/views/slider/src/main/java/org/apache/ambari/view/slider/SliderAppsViewControllerImpl.java > > It includes scale-out/in an existing Slider Application. > > BTW, the following doc describes REST API of Slider. However I never try > REST API directly but SliderClient class as I mentioned above. > > http://slider.incubator.apache.org/docs/api/slider_REST_api_v2.html > > Thanks, > > Yong > > > > On Fri, Jul 24, 2015 at 5:35 PM, Jiwon Seo <[email protected]> wrote: > >> Thank you for your reply. >> >> As far as I understand, "slider flex ... " in the link >> http://slider.incubator.apache.org/docs/manpage.html performs scaling out >> upon the execution of the command. My questions is, does Slider have a >> programmable interface for scaling out? (and I'm not talking about >> automatic scaling, but about the functionality for an application to >> request more machines -- or memory/cpus from Slider). >> >> Does Slider have such interface (API)? >> >> Thanks! >> >> -Jiwon >> >> >> On Fri, Jul 24, 2015 at 1:51 PM, Yong Feng <[email protected]> wrote: >> >>> Hi Jiwon, >>> >>> Slider supports horizontally scale out/in your application. Please refer >> to >>> following link for the user doc of slider commands which describes >> "slider >>> flex ..." >>> >>> http://slider.incubator.apache.org/docs/manpage.html >>> >>> Besides, YARN 1197 plans to support resizing YARN container. By >> leveraging >>> it, Slider is able to support vertically scale up/down your application >>> instance, for example increasing memory from 2GB to 4GB. >>> >>> https://issues.apache.org/jira/browse/YARN-1197 >>> >>> Based on slider's capabilities of scale out/in and scale up/down, you >> could >>> develop a driver to automatically flex your application. As I know there >> is >>> a team who is working on an open source general auto-scaler for >> YARN/Slider >>> based application which is named as Jaguar. You may contact them for >>> details. >>> >>> >>> >> https://hadoopsummit.uservoice.com/forums/283262-hadoop-governance-security-operations/suggestions/7074316-project-jaguar-an-application-autoscaler-for-hado >>> >>> Please feel free to let me know if you have any question. >>> >>> Thanks, >>> >>> Yong >>> >>> >>> On Fri, Jul 24, 2015 at 2:24 PM, Jiwon Seo <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> I'm working on a research system for in-memory graph processing (called >>>> SociaLite). >>>> >>>> I'm curious if Slider supports dynamic resizing, requested by >>> applications. >>>> That is, if the application detects memory pressure, the it asks Slider >>> for >>>> more resources (e.g. nodes with XXX mb memory); then Slider will tell >> the >>>> application how many nodes are allocated/assigned to the app. >>>> >>>> Does Slider have this kind of application interface? (if so, any link >> to >>>> the doc?) Or for that functionality, should I directly use Yarn API? >>>> >>>> Thanks, >>>> >>>> -Jiwon >>>> >>> >>
