Addressed review comments from Yuliya and updated the PR: https://github.com/mesos/myriad/pull/129
@Jim - Your UI expertise is much sought to support "constraints" parameter in both flexup/down and "profile" in flexdown. The API doc is updated as part of the PR: https://github.com/mesos/myriad/pull/129/files#diff-96cc378dcdef180641176501f42183c7 Thanks, Santosh On Mon, Sep 14, 2015 at 10:54 AM, Santosh Marella <[email protected]> wrote: > The proposal is to have "constraints" parameter supported for both > flexup/flexdown. > So, one could flexdown NMs on specific hosts, for e..g by providing a > constraint > around "hostname" (/api/flexdown {"constraints": [["hostname" LIKE " > specific.host.com"]]}. > > Santosh > > On Mon, Sep 14, 2015 at 2:31 AM, Adam Bordelon <[email protected]> wrote: > >> +1, but EQUALS might be an easier starting point than LIKE. Your choice. >> What about flexing down specific hosts? Is that out of scope for now? >> >> On Wed, Sep 2, 2015 at 2:13 PM, Santosh Marella <[email protected]> >> wrote: >> >> > Ping! Checking if there is anymore feedback on the APIs for flexup/down. >> > >> > Here is a textual summary of what's being proposed: >> > 1. Add a new “constraints” parameter in both flexup and flexdown APIs >> (just >> > like how Marathon does). >> > Initially, start with support for LIKE operator in "constraints" that >> > operates on “hostname” and “mesos slave attribute >> > <http://mesos.apache.org/documentation/attributes-resources/>” fields >> > using >> > regex. >> > 2. Add a new “profile” parameter in the flexdown API. >> > >> > #1 would give admins a better control over where the NMs should be >> launched >> > or which specific NMs should be shut down. >> > #2 helps with shutting down multiple NMs of same "profile" in one shot. >> > >> > If the above sounds reasonable to start with, I'd like to start working >> on >> > the code changes. >> > >> > Thanks, >> > Santosh >> > >> > On Thu, Aug 27, 2015 at 4:25 PM, Santosh Marella <[email protected] >> > >> > wrote: >> > >> > > > One additional thing I'd like is the ability to flexup a nmnode on >> all >> > > > agents with a certain attribute as opposed to a fixed number. >> > > >> > > Certainly! Marathon places tasks on agents sharing a common attribute. >> > > Myriad could/should do the same for placing NMs. If each agent running >> > > HDFS data node shares an attribute such as "dfs", then NMs could >> always >> > > be launched on the "dfs" nodes. This improves data locality for YARN >> jobs >> > > dramatically. >> > > >> > > Santosh >> > > >> > > On Thu, Aug 27, 2015 at 2:47 PM, Darin Johnson < >> [email protected]> >> > > wrote: >> > > >> > >> One additional thing I'd like is the ability to flexup a nmnode on >> all >> > >> agents with a certain attribute as opposed to a fixed number. >> > >> >> > > >> > > >> > >> >> > >> Also, when I get back from vacation I plan on scoping mesos-1739 >> > (dynamic >> > >> attributes) which would allow for tighter integration with the hdfs >> > >> framework. An alternative would be to get hostnames from the >> namenode, >> > >> though not as seemless. >> > >> (Reviving this thread) >> > >> >> > >> We've discussed several great points in the thread (PUT vs POST, need >> > for >> > >> GET, JSON payload vs parameters in URL, declarative interface etc). >> > >> Just to get us going, I think we should focus on a couple of things >> that >> > >> will be useful for Myriad users, while leaving them flexible enough >> to >> > be >> > >> evolved in the future. >> > >> >> > >> What I heard from several folks (some of it brought up again at >> > MesosCon) >> > >> about the flex up/down APIs is this: >> > >> - flexup doesn't support launching NMs on specific set of hosts. >> This is >> > >> especially needed to launch NMs on same set of nodes that have HDFS >> > >> DataNode running. >> > >> - flexdown lacks an option to shut down NMs with a specific profile. >> > >> Today, >> > >> we bring down ANY arbitrary NM. >> > >> - flexdown lacks an option to shutdown NMs running on specific hosts. >> > >> >> > >> I captured my thoughts in a document here: >> > >> >> > >> >> > >> https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t >> > >> >> > >> @Jim: Happy to collaborate at one single place (Swagger/Google Doc) >> to >> > >> finalize the APIs. Just let me know. >> > >> >> > >> Thanks, >> > >> Santosh >> > >> >> > >> On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <[email protected]> >> wrote: >> > >> >> > >> > Seems like POST is a winner with people. >> > >> > >> > >> > Another thing to consider is how we want the REST interface to be >> vs >> > >> what >> > >> > we want the UI to do. The UI could support flexup/flexdown like it >> is >> > >> while >> > >> > the REST interface is just a declarative state like Adam suggested. >> > The >> > >> UI >> > >> > would just be responsible for translating the request into the new >> > >> state. >> > >> > >> > >> > Tomorrow I'll try to put together another swagger doc with some of >> the >> > >> > suggested options. >> > >> > >> > >> > >> > >> > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman >> > >> > <[email protected] >> > >> > > wrote: >> > >> > >> > >> > > I think we are at the point to list all the options we want >> "flex" >> > API >> > >> to >> > >> > > support. >> > >> > > 1. Do we continue supporting flexup/down or just "flex" with >> > >> additional >> > >> > > "preposition" like up/down:https://hostname:port/flex/up(down) >> > >> > > 2. I think we should switch to POST and may be maintain PUT for >> > legacy >> > >> > (if >> > >> > > even needed to keep it). We are not DB after all and not storing >> any >> > >> > > retrievable info here :) >> > >> > > 3. We need to add status (GET) to see the status - though I >> think we >> > >> have >> > >> > > one >> > >> > > 4. Define JSON payload to support different cases a. providing >> > >> > > different profiles together: [{profile:"big", >> > >> > > instances:2},{profile:"medium",instances:6}] b. provide what >> > state >> > >> we >> > >> > > want Myriad to be in: "I want 10 medium instances" and then >> Myriad >> > >> will >> > >> > do >> > >> > > whatever isnecessary to transition to that state, >> > >> > adding/removing/resizing >> > >> > > NMs" c. flex/down particular instance IDs d. flex up/down >> > >> preferred >> > >> > > hosts, delays, others >> > >> > > 5. How all this fits into FineGrain Scaling? With it we would do >> > >> > automatic >> > >> > > flex up/down. And the less knobs admin will have to turn the >> easier >> > it >> > >> is >> > >> > > for admin and the end users. >> > >> > > >> > >> > > From: Adam Bordelon <[email protected]> >> > >> > > To: [email protected] >> > >> > > Sent: Sunday, June 14, 2015 2:54 PM >> > >> > > Subject: Re: Flex API >> > >> > > >> > >> > > (In addition,) I'd also like to see a more declarative interface. >> > >> Instead >> > >> > > of "add two more instances", the user(s) could just specify the >> > >> desired >> > >> > > state of "I want 10 medium instances" and then Myriad will do >> > whatever >> > >> is >> > >> > > necessary to transition to that state, adding/removing/resizing >> NMs >> > as >> > >> > > necessary. >> > >> > > >> > >> > > >> > >> > > >> > >> > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena < >> > >> > > [email protected] >> > >> > > > wrote: >> > >> > > >> > >> > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <[email protected]> >> > >> wrote: >> > >> > > > >> > >> > > > > What verb to use when outside of database land can be >> argued. I >> > >> would >> > >> > > > vote >> > >> > > > > for POST over PUT just because I tend to default to POST. PUT >> > was >> > >> > there >> > >> > > > > when I showed up, so I left it. >> > >> > > > >> > >> > > > >> > >> > > > Last time I agonized about PUT vs POST the most logical >> > distinction >> > >> I >> > >> > > found >> > >> > > > was that PUT should be used for idempotent operations, while >> POST >> > >> for >> > >> > > > non-idempotent (like we have here with flex-up, since >> instance-ids >> > >> are >> > >> > > > generated). >> > >> > > > >> > >> > > > Since the api doesn't wait until the >> > >> > > > > instances are created to return, we can't really return the >> > >> instance >> > >> > > IDs >> > >> > > > we >> > >> > > > > created. >> > >> > > > > >> > >> > > > >> > >> > > > That seems OK to me. >> > >> > > > >> > >> > > > >> > >> > > > > The GET would just return some status? >> > >> > > > > >> > >> > > > >> > >> > > > Yeah, I was thinking that this would be needed for a future GUI >> > >> where >> > >> > we >> > >> > > > list all instances with parameters and status for each >> (profile, >> > >> > current >> > >> > > > cpu/ram/disk, node, uptime). I'm picturing checkboxes next to >> > each >> > >> so >> > >> > > > users can multi-select and hit 'delete' to wipe them away (like >> > >> > flex-down >> > >> > > > does now). >> > >> > > > >> > >> > > > The PATCH is interesting >> > >> > > > > >> > >> > > > >> > >> > > > Yeah, I started to write PUT but to REST geeks PUT implies you >> > >> always >> > >> > > have >> > >> > > > to rewrite the complete object when making changes. PATCH >> allows >> > >> more >> > >> > > > flexible modifications. >> > >> > > > >> > >> > > > The DELETE makes sense to me. >> > >> > > > > >> > >> > > > > Your use of instances vs instance is interesting. Perhaps we >> > want >> > >> to >> > >> > > > > support POST [{profile:"big", instances:2},{profile:"medium", >> > >> > > > instances:6}] >> > >> > > > > >> > >> > > > >> > >> > > > Yeah, that'd be cool! >> > >> > > > >> > >> > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena < >> > >> > > > > [email protected] >> > >> > > > > > wrote: >> > >> > > > > >> > >> > > > > > Any reason we need to make it action-based >> (flex-up/flex-down) >> > >> > rather >> > >> > > > > than >> > >> > > > > > instance outcome-based? The way i've seen similar things >> > >> modeled >> > >> > in >> > >> > > > > other >> > >> > > > > > REST APIs is similar to below. This may also fit better >> into >> > >> REST >> > >> > > > client >> > >> > > > > > frameworks. >> > >> > > > > > >> > >> > > > > > Create - POST /instances/ {"profile" : "zero", "instances" >> : >> > 2, >> > >> > > > "delay" : >> > >> > > > > > 2, "preferredHosts" : ["host1", "host2"]} >> > >> > > > > > Read - GET /instances/ or GET /instances/<instanceid> >> > >> > > > > > Update (resize?) - PATCH /instances/<instanceid> >> {"profile" : >> > >> > "big"} >> > >> > > > > > Delete - DELETE /instances/<instanceid> >> > >> > > > > > >> > >> > > > > > Thoughts? >> > >> > > > > > >> > >> > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar < >> [email protected] >> > > >> > >> > > wrote: >> > >> > > > > > >> > >> > > > > > > That is always an issue with parameters in the URL, and >> > >> there's >> > >> > > > nothing >> > >> > > > > > > worse than parameters in the URL and having to put JSON >> in >> > the >> > >> > body >> > >> > > > > > because >> > >> > > > > > > we want to extend it later. Currently flexup takes a JSON >> > blob >> > >> in >> > >> > > the >> > >> > > > > > body >> > >> > > > > > > which we could add more to like you showed. I guess the >> > >> question >> > >> > is >> > >> > > > if >> > >> > > > > we >> > >> > > > > > > want to support a quick and dirty flexup api via URL >> > >> parameters. >> > >> > I >> > >> > > > > > wouldn't >> > >> > > > > > > go any further than these parameters, but all this would >> do >> > at >> > >> > this >> > >> > > > > point >> > >> > > > > > > is make testing with a curl command easier. Getting rid >> of >> > it >> > >> for >> > >> > > > > flexup >> > >> > > > > > is >> > >> > > > > > > fine with me. >> > >> > > > > > > >> > >> > > > > > > I do like the flexdown/instance/{instance-id} though. We >> > >> should >> > >> > > > > probably >> > >> > > > > > do >> > >> > > > > > > one with flexdown/instance with a json array of >> instance-ids >> > >> too >> > >> > > > > though. >> > >> > > > > > > >> > >> > > > > > > Also with the parameter verification patch I just >> submitted, >> > >> if >> > >> > you >> > >> > > > ask >> > >> > > > > > to >> > >> > > > > > > flexdown 10 instances and you only have 5 running it will >> > just >> > >> > log >> > >> > > a >> > >> > > > > > > warning and kill all 5. Not sure if that is the desired >> > >> behavior. >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella < >> > >> > > > > [email protected]> >> > >> > > > > > > wrote: >> > >> > > > > > > >> > >> > > > > > > > I think rather than having an API defined as >> > >> > > > > > > > "/cluster/flexup/profile/{profile-name}", it's better >> to >> > >> define >> > >> > > it >> > >> > > > > just >> > >> > > > > > > as >> > >> > > > > > > > "/cluster/flexup" and have parameters such as "profile: >> > >> > > > > > > > <a-value-for-profile-name>" in the JSON payload. The >> > reason >> > >> is, >> > >> > > if >> > >> > > > > the >> > >> > > > > > > > params are added into the API endpoint, it becomes less >> > >> > flexible >> > >> > > to >> > >> > > > > > > evolve. >> > >> > > > > > > > For e.g. /cluster/flexup currently can take just >> "profile" >> > >> and >> > >> > > > > > > "instances" >> > >> > > > > > > > in it's payload, but in the future can optionally take >> > >> > parameters >> > >> > > > > such >> > >> > > > > > as >> > >> > > > > > > > "preferredHosts: [host1, host2, host9]", "delay: >> > >> > > > > > > > <take_some_sweet_time_before_flexing_up>" etc. >> > >> > > > > > > > >> > >> > > > > > > > Thoughts? >> > >> > > > > > > > >> > >> > > > > > > > Santosh >> > >> > > > > > > > >> > >> > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar < >> > >> [email protected]> >> > >> > > > > wrote: >> > >> > > > > > > > >> > >> > > > > > > > > I thought that might happen. I created a gist here: >> > >> > > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e >> > >> > > > > > > > > >> > >> > > > > > > > > And just in case.... >> > >> > > > > > > > > >> > >> > > > > > > > > swagger: '2.0' >> > >> > > > > > > > > info: >> > >> > > > > > > > > version: 0.0.1 >> > >> > > > > > > > > title: Myriad API >> > >> > > > > > > > > description: | >> > >> > > > > > > > > Myriad API description >> > >> > > > > > > > > basePath: /api >> > >> > > > > > > > > schemes: >> > >> > > > > > > > > - http >> > >> > > > > > > > > consumes: >> > >> > > > > > > > > - application/json >> > >> > > > > > > > > - text/plain >> > >> > > > > > > > > produces: >> > >> > > > > > > > > - application/json >> > >> > > > > > > > > - text/plain >> > >> > > > > > > > > paths: >> > >> > > > > > > > > /cluster/flexup: >> > >> > > > > > > > > put: >> > >> > > > > > > > > parameters: >> > >> > > > > > > > > - name: instance >> > >> > > > > > > > > in: body >> > >> > > > > > > > > description: Instance profile and quantity >> to >> > >> flex >> > >> > up >> > >> > > > > > > > > schema: >> > >> > > > > > > > > $ref: '#/definitions/FlexUp' >> > >> > > > > > > > > required: true >> > >> > > > > > > > > description: Original Flexup API >> > >> > > > > > > > > responses: >> > >> > > > > > > > > 200: >> > >> > > > > > > > > description: OK >> > >> > > > > > > > > /cluster/flexup/profile/{profile-name}: >> > >> > > > > > > > > put: >> > >> > > > > > > > > parameters: >> > >> > > > > > > > > - name: profile-name >> > >> > > > > > > > > in: path >> > >> > > > > > > > > type: string >> > >> > > > > > > > > description: Instance profile name >> > >> > > > > > > > > description: Flexup a single instance of a >> > specified >> > >> > > > profile >> > >> > > > > > > > > responses: >> > >> > > > > > > > > 200: >> > >> > > > > > > > > description: OK >> > >> > > > > > > > > /cluster/flexup/profile/{profile-name}/{instances}: >> > >> > > > > > > > > put: >> > >> > > > > > > > > parameters: >> > >> > > > > > > > > - name: profile-name >> > >> > > > > > > > > in: path >> > >> > > > > > > > > type: string >> > >> > > > > > > > > description: Instance profile name >> > >> > > > > > > > > - name: instances >> > >> > > > > > > > > in: path >> > >> > > > > > > > > type: integer >> > >> > > > > > > > > description: Number of instances >> > >> > > > > > > > > description: Flexup a multiple instances of a >> > >> specified >> > >> > > > > profile >> > >> > > > > > > > > responses: >> > >> > > > > > > > > 200: >> > >> > > > > > > > > description: OK >> > >> > > > > > > > > /cluster/flexdown: >> > >> > > > > > > > > put: >> > >> > > > > > > > > parameters: >> > >> > > > > > > > > - name: instance >> > >> > > > > > > > > in: body >> > >> > > > > > > > > description: Number of instances to flex >> down >> > >> > > > > > > > > schema: >> > >> > > > > > > > > $ref: '#/definitions/FlexDown' >> > >> > > > > > > > > required: true >> > >> > > > > > > > > description: Original Flexdown API >> > >> > > > > > > > > responses: >> > >> > > > > > > > > 200: >> > >> > > > > > > > > description: OK >> > >> > > > > > > > > /cluster/flexdown/profile/{profile-name}: >> > >> > > > > > > > > put: >> > >> > > > > > > > > parameters: >> > >> > > > > > > > > - name: profile-name >> > >> > > > > > > > > in: path >> > >> > > > > > > > > type: string >> > >> > > > > > > > > description: Instance profile name >> > >> > > > > > > > > description: Flexup a single instance of a >> > specified >> > >> > > > profile >> > >> > > > > > > > > responses: >> > >> > > > > > > > > 200: >> > >> > > > > > > > > description: OK >> > >> > > > > > > > > >> /cluster/flexdown/profile/{profile-name}/{instances}: >> > >> > > > > > > > > put: >> > >> > > > > > > > > parameters: >> > >> > > > > > > > > - name: profile-name >> > >> > > > > > > > > in: path >> > >> > > > > > > > > type: string >> > >> > > > > > > > > description: Instance profile name >> > >> > > > > > > > > - name: instances >> > >> > > > > > > > > in: path >> > >> > > > > > > > > type: integer >> > >> > > > > > > > > description: Number of instances >> > >> > > > > > > > > description: Flexdown a multiple instances of a >> > >> > specified >> > >> > > > > > profile >> > >> > > > > > > > > responses: >> > >> > > > > > > > > 200: >> > >> > > > > > > > > description: OK >> > >> > > > > > > > > /cluster/flexdown/instance/{instance-id}: >> > >> > > > > > > > > put: >> > >> > > > > > > > > parameters: >> > >> > > > > > > > > - name: instance-id >> > >> > > > > > > > > in: path >> > >> > > > > > > > > type: string >> > >> > > > > > > > > description: Instance profile name >> > >> > > > > > > > > description: Flexup a single instance of a >> > specified >> > >> > > > profile >> > >> > > > > > > > > responses: >> > >> > > > > > > > > 200: >> > >> > > > > > > > > description: OK >> > >> > > > > > > > > definitions: >> > >> > > > > > > > > FlexUp: >> > >> > > > > > > > > properties: >> > >> > > > > > > > > profile: >> > >> > > > > > > > > type: string >> > >> > > > > > > > > instances: >> > >> > > > > > > > > type: integer >> > >> > > > > > > > > format: int32 >> > >> > > > > > > > > FlexDown: >> > >> > > > > > > > > properties: >> > >> > > > > > > > > instances: >> > >> > > > > > > > > type: integer >> > >> > > > > > > > > format: int32 >> > >> > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella < >> > >> > > > > > > [email protected]> >> > >> > > > > > > > > wrote: >> > >> > > > > > > > > >> > >> > > > > > > > > > Hi Jim, >> > >> > > > > > > > > > >> > >> > > > > > > > > > Did you attach a file? I think the apache mailing >> > list >> > >> > > > > swallowed >> > >> > > > > > > the >> > >> > > > > > > > > > attachment. Can you please send us the document >> > contents >> > >> in >> > >> > > > plain >> > >> > > > > > > text? >> > >> > > > > > > > > > >> > >> > > > > > > > > > Thanks, >> > >> > > > > > > > > > Santosh >> > >> > > > > > > > > > >> > >> > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar < >> > >> > > [email protected]> >> > >> > > > > > > wrote: >> > >> > > > > > > > > > >> > >> > > > > > > > > > > This is in regards to >> > >> > > > > https://github.com/mesos/myriad/issues/89 >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > I created a swagger document (attached) of the >> > current >> > >> > flex >> > >> > > > > APIs >> > >> > > > > > > and >> > >> > > > > > > > > what >> > >> > > > > > > > > > > I propose for the new flex apis. If people like >> > >> swagger, >> > >> > I >> > >> > > > can >> > >> > > > > > > > document >> > >> > > > > > > > > > the >> > >> > > > > > > > > > > rest of the API, including the correct response >> > codes. >> > >> > > Either >> > >> > > > > > way I >> > >> > > > > > > > > think >> > >> > > > > > > > > > > we should discuss what the API should be going >> > >> forward. >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > Paste the file contents into >> > >> http://editor.swagger.io/ >> > >> > and >> > >> > > > > > you'll >> > >> > > > > > > > get >> > >> > > > > > > > > > > nice HTML to browse. >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > >> > >> >> > > >> > > >> > >> > >
