Can this be equally solved with the Yarn capacity or fairscheduler? You may want to add another scheduler having some more intelligence, although this is probably also an interesting research problem
Le mar. 18 août 2015 à 5:35, YONG FENG (JIRA) <[email protected]> a écrit : > > [ > https://issues.apache.org/jira/browse/SLIDER-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700680#comment-14700680 > ] > > YONG FENG commented on SLIDER-868: > ---------------------------------- > > Thanks Gour for the reply. > > The Slider always need negotiate resources with YARN resource manager > during initially creating an application, flexing up and down an > application ... If we know the workload pattern of an application and want > to make sure it is able to get enough resource for peak time while free > resources for other applications during idle time. We should enhance YARN > to support time-windows based resource plan, instead of in Slider as it is > YARN who control the deserved resources. Even though you have some logic in > Slider and try to allocate resource in advance, it still cannot guarantee > enough resource for the application and guarantee the SLA. It is actually > similar as an admin issue flex up request in advance. > > I think you make a good point to plan resource among applications better > based on known application workload pattern/requriment. However the place > to suppport the dynamic resource plan is actually in YARN instead of Slider. > > So YARN has time-windows based resource plan to grant more deserved > resource to an application during its peak time, and lower down its > deserved resource duirng its idle time to balance the quota. While Slider > orchestrate the flow of creating application, flexing up/down application, > stop application and so on. An App manager automatically control the size > of applcation based on time-windows based policy or metrics based policy. > > Does it make sense? > > Thanks, > > Yong > > > Ability to put a Slider application on cruise control > > ----------------------------------------------------- > > > > Key: SLIDER-868 > > URL: https://issues.apache.org/jira/browse/SLIDER-868 > > Project: Slider > > Issue Type: New Feature > > Components: agent, app-package, appmaster > > Affects Versions: Slider 0.70 > > Reporter: Gour Saha > > > > You create a Slider application package, deploy it to a YARN cluster and > manage it. From the management perspective it is primarily flexing. Based > on needs (and the architecture of the application) you grow or shrink > specific components of your long running application from time to time. Of > course you can set some constraints like affinity, anti-affinity, and > strict placement (for data locality or other reasons). Some of these are > handled very well by Slider, others are best efforts. > > However long running applications have an inherent need to be auto (or > even self) managed. This can be achieved by a custom management tool, > interacting with Slider client based on constant feedback on the health of > the application (metrics, alerts, etc.). This is primarily reactive > management. > > There is also proactive management, where the application owner is aware > of the usage pattern of the application over time. For example, a financial > application usage peaks between 8am to 4pm Mon to Sat (local time), and > slows down at other times. A tax application usage peaks for a few months > prior to April 15 and then slows down for several months. Certain > healthcare applications peak during flu season. You get the point! > > It should be possible to declaratively define such an application usage > skyline, which can be fed to Slider and put an application on cruise > control. The specification can be modified dynamically and Slider should > honor the modified version for (reasonably acceptable) future state of the > application. > > This kind of feature would need support from YARN. There should be a way > for Slider to provide details to YARN for guaranteed future capacity > planning. > > Note: It is the negotiation that Slider will do with YARN, to ensure the > guaranteed (or best effort) future capacity planning to maintain the > skyline is what makes this pro-active management useful. Nothing stops an > application owner to write pro-active tools to manage a skyline. > > > > -- > This message was sent by Atlassian JIRA > (v6.3.4#6332) >
