The proposal is to have "constraints" parameter supported for both
flexup/flexdown.
So, one could flexdown NMs on specific hosts, for e..g by providing a
constraint
around "hostname" (/api/flexdown {"constraints": [["hostname" LIKE "
specific.host.com"]]}.Santosh On Mon, Sep 14, 2015 at 2:31 AM, Adam Bordelon <[email protected]> wrote: > +1, but EQUALS might be an easier starting point than LIKE. Your choice. > What about flexing down specific hosts? Is that out of scope for now? > > On Wed, Sep 2, 2015 at 2:13 PM, Santosh Marella <[email protected]> > wrote: > > > Ping! Checking if there is anymore feedback on the APIs for flexup/down. > > > > Here is a textual summary of what's being proposed: > > 1. Add a new “constraints” parameter in both flexup and flexdown APIs > (just > > like how Marathon does). > > Initially, start with support for LIKE operator in "constraints" that > > operates on “hostname” and “mesos slave attribute > > <http://mesos.apache.org/documentation/attributes-resources/>” fields > > using > > regex. > > 2. Add a new “profile” parameter in the flexdown API. > > > > #1 would give admins a better control over where the NMs should be > launched > > or which specific NMs should be shut down. > > #2 helps with shutting down multiple NMs of same "profile" in one shot. > > > > If the above sounds reasonable to start with, I'd like to start working > on > > the code changes. > > > > Thanks, > > Santosh > > > > On Thu, Aug 27, 2015 at 4:25 PM, Santosh Marella <[email protected]> > > wrote: > > > > > > One additional thing I'd like is the ability to flexup a nmnode on > all > > > > agents with a certain attribute as opposed to a fixed number. > > > > > > Certainly! Marathon places tasks on agents sharing a common attribute. > > > Myriad could/should do the same for placing NMs. If each agent running > > > HDFS data node shares an attribute such as "dfs", then NMs could always > > > be launched on the "dfs" nodes. This improves data locality for YARN > jobs > > > dramatically. > > > > > > Santosh > > > > > > On Thu, Aug 27, 2015 at 2:47 PM, Darin Johnson < > [email protected]> > > > wrote: > > > > > >> One additional thing I'd like is the ability to flexup a nmnode on all > > >> agents with a certain attribute as opposed to a fixed number. > > >> > > > > > > > > >> > > >> Also, when I get back from vacation I plan on scoping mesos-1739 > > (dynamic > > >> attributes) which would allow for tighter integration with the hdfs > > >> framework. An alternative would be to get hostnames from the namenode, > > >> though not as seemless. > > >> (Reviving this thread) > > >> > > >> We've discussed several great points in the thread (PUT vs POST, need > > for > > >> GET, JSON payload vs parameters in URL, declarative interface etc). > > >> Just to get us going, I think we should focus on a couple of things > that > > >> will be useful for Myriad users, while leaving them flexible enough to > > be > > >> evolved in the future. > > >> > > >> What I heard from several folks (some of it brought up again at > > MesosCon) > > >> about the flex up/down APIs is this: > > >> - flexup doesn't support launching NMs on specific set of hosts. This > is > > >> especially needed to launch NMs on same set of nodes that have HDFS > > >> DataNode running. > > >> - flexdown lacks an option to shut down NMs with a specific profile. > > >> Today, > > >> we bring down ANY arbitrary NM. > > >> - flexdown lacks an option to shutdown NMs running on specific hosts. > > >> > > >> I captured my thoughts in a document here: > > >> > > >> > > > https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t > > >> > > >> @Jim: Happy to collaborate at one single place (Swagger/Google Doc) to > > >> finalize the APIs. Just let me know. > > >> > > >> Thanks, > > >> Santosh > > >> > > >> On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <[email protected]> wrote: > > >> > > >> > Seems like POST is a winner with people. > > >> > > > >> > Another thing to consider is how we want the REST interface to be vs > > >> what > > >> > we want the UI to do. The UI could support flexup/flexdown like it > is > > >> while > > >> > the REST interface is just a declarative state like Adam suggested. > > The > > >> UI > > >> > would just be responsible for translating the request into the new > > >> state. > > >> > > > >> > Tomorrow I'll try to put together another swagger doc with some of > the > > >> > suggested options. > > >> > > > >> > > > >> > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman > > >> > <[email protected] > > >> > > wrote: > > >> > > > >> > > I think we are at the point to list all the options we want "flex" > > API > > >> to > > >> > > support. > > >> > > 1. Do we continue supporting flexup/down or just "flex" with > > >> additional > > >> > > "preposition" like up/down:https://hostname:port/flex/up(down) > > >> > > 2. I think we should switch to POST and may be maintain PUT for > > legacy > > >> > (if > > >> > > even needed to keep it). We are not DB after all and not storing > any > > >> > > retrievable info here :) > > >> > > 3. We need to add status (GET) to see the status - though I think > we > > >> have > > >> > > one > > >> > > 4. Define JSON payload to support different cases a. providing > > >> > > different profiles together: [{profile:"big", > > >> > > instances:2},{profile:"medium",instances:6}] b. provide what > > state > > >> we > > >> > > want Myriad to be in: "I want 10 medium instances" and then Myriad > > >> will > > >> > do > > >> > > whatever isnecessary to transition to that state, > > >> > adding/removing/resizing > > >> > > NMs" c. flex/down particular instance IDs d. flex up/down > > >> preferred > > >> > > hosts, delays, others > > >> > > 5. How all this fits into FineGrain Scaling? With it we would do > > >> > automatic > > >> > > flex up/down. And the less knobs admin will have to turn the > easier > > it > > >> is > > >> > > for admin and the end users. > > >> > > > > >> > > From: Adam Bordelon <[email protected]> > > >> > > To: [email protected] > > >> > > Sent: Sunday, June 14, 2015 2:54 PM > > >> > > Subject: Re: Flex API > > >> > > > > >> > > (In addition,) I'd also like to see a more declarative interface. > > >> Instead > > >> > > of "add two more instances", the user(s) could just specify the > > >> desired > > >> > > state of "I want 10 medium instances" and then Myriad will do > > whatever > > >> is > > >> > > necessary to transition to that state, adding/removing/resizing > NMs > > as > > >> > > necessary. > > >> > > > > >> > > > > >> > > > > >> > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena < > > >> > > [email protected] > > >> > > > wrote: > > >> > > > > >> > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <[email protected]> > > >> wrote: > > >> > > > > > >> > > > > What verb to use when outside of database land can be argued. > I > > >> would > > >> > > > vote > > >> > > > > for POST over PUT just because I tend to default to POST. PUT > > was > > >> > there > > >> > > > > when I showed up, so I left it. > > >> > > > > > >> > > > > > >> > > > Last time I agonized about PUT vs POST the most logical > > distinction > > >> I > > >> > > found > > >> > > > was that PUT should be used for idempotent operations, while > POST > > >> for > > >> > > > non-idempotent (like we have here with flex-up, since > instance-ids > > >> are > > >> > > > generated). > > >> > > > > > >> > > > Since the api doesn't wait until the > > >> > > > > instances are created to return, we can't really return the > > >> instance > > >> > > IDs > > >> > > > we > > >> > > > > created. > > >> > > > > > > >> > > > > > >> > > > That seems OK to me. > > >> > > > > > >> > > > > > >> > > > > The GET would just return some status? > > >> > > > > > > >> > > > > > >> > > > Yeah, I was thinking that this would be needed for a future GUI > > >> where > > >> > we > > >> > > > list all instances with parameters and status for each (profile, > > >> > current > > >> > > > cpu/ram/disk, node, uptime). I'm picturing checkboxes next to > > each > > >> so > > >> > > > users can multi-select and hit 'delete' to wipe them away (like > > >> > flex-down > > >> > > > does now). > > >> > > > > > >> > > > The PATCH is interesting > > >> > > > > > > >> > > > > > >> > > > Yeah, I started to write PUT but to REST geeks PUT implies you > > >> always > > >> > > have > > >> > > > to rewrite the complete object when making changes. PATCH > allows > > >> more > > >> > > > flexible modifications. > > >> > > > > > >> > > > The DELETE makes sense to me. > > >> > > > > > > >> > > > > Your use of instances vs instance is interesting. Perhaps we > > want > > >> to > > >> > > > > support POST [{profile:"big", instances:2},{profile:"medium", > > >> > > > instances:6}] > > >> > > > > > > >> > > > > > >> > > > Yeah, that'd be cool! > > >> > > > > > >> > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena < > > >> > > > > [email protected] > > >> > > > > > wrote: > > >> > > > > > > >> > > > > > Any reason we need to make it action-based > (flex-up/flex-down) > > >> > rather > > >> > > > > than > > >> > > > > > instance outcome-based? The way i've seen similar things > > >> modeled > > >> > in > > >> > > > > other > > >> > > > > > REST APIs is similar to below. This may also fit better > into > > >> REST > > >> > > > client > > >> > > > > > frameworks. > > >> > > > > > > > >> > > > > > Create - POST /instances/ {"profile" : "zero", "instances" : > > 2, > > >> > > > "delay" : > > >> > > > > > 2, "preferredHosts" : ["host1", "host2"]} > > >> > > > > > Read - GET /instances/ or GET /instances/<instanceid> > > >> > > > > > Update (resize?) - PATCH /instances/<instanceid> {"profile" > : > > >> > "big"} > > >> > > > > > Delete - DELETE /instances/<instanceid> > > >> > > > > > > > >> > > > > > Thoughts? > > >> > > > > > > > >> > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar < > [email protected] > > > > > >> > > wrote: > > >> > > > > > > > >> > > > > > > That is always an issue with parameters in the URL, and > > >> there's > > >> > > > nothing > > >> > > > > > > worse than parameters in the URL and having to put JSON in > > the > > >> > body > > >> > > > > > because > > >> > > > > > > we want to extend it later. Currently flexup takes a JSON > > blob > > >> in > > >> > > the > > >> > > > > > body > > >> > > > > > > which we could add more to like you showed. I guess the > > >> question > > >> > is > > >> > > > if > > >> > > > > we > > >> > > > > > > want to support a quick and dirty flexup api via URL > > >> parameters. > > >> > I > > >> > > > > > wouldn't > > >> > > > > > > go any further than these parameters, but all this would > do > > at > > >> > this > > >> > > > > point > > >> > > > > > > is make testing with a curl command easier. Getting rid of > > it > > >> for > > >> > > > > flexup > > >> > > > > > is > > >> > > > > > > fine with me. > > >> > > > > > > > > >> > > > > > > I do like the flexdown/instance/{instance-id} though. We > > >> should > > >> > > > > probably > > >> > > > > > do > > >> > > > > > > one with flexdown/instance with a json array of > instance-ids > > >> too > > >> > > > > though. > > >> > > > > > > > > >> > > > > > > Also with the parameter verification patch I just > submitted, > > >> if > > >> > you > > >> > > > ask > > >> > > > > > to > > >> > > > > > > flexdown 10 instances and you only have 5 running it will > > just > > >> > log > > >> > > a > > >> > > > > > > warning and kill all 5. Not sure if that is the desired > > >> behavior. > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella < > > >> > > > > [email protected]> > > >> > > > > > > wrote: > > >> > > > > > > > > >> > > > > > > > I think rather than having an API defined as > > >> > > > > > > > "/cluster/flexup/profile/{profile-name}", it's better to > > >> define > > >> > > it > > >> > > > > just > > >> > > > > > > as > > >> > > > > > > > "/cluster/flexup" and have parameters such as "profile: > > >> > > > > > > > <a-value-for-profile-name>" in the JSON payload. The > > reason > > >> is, > > >> > > if > > >> > > > > the > > >> > > > > > > > params are added into the API endpoint, it becomes less > > >> > flexible > > >> > > to > > >> > > > > > > evolve. > > >> > > > > > > > For e.g. /cluster/flexup currently can take just > "profile" > > >> and > > >> > > > > > > "instances" > > >> > > > > > > > in it's payload, but in the future can optionally take > > >> > parameters > > >> > > > > such > > >> > > > > > as > > >> > > > > > > > "preferredHosts: [host1, host2, host9]", "delay: > > >> > > > > > > > <take_some_sweet_time_before_flexing_up>" etc. > > >> > > > > > > > > > >> > > > > > > > Thoughts? > > >> > > > > > > > > > >> > > > > > > > Santosh > > >> > > > > > > > > > >> > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar < > > >> [email protected]> > > >> > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > I thought that might happen. I created a gist here: > > >> > > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e > > >> > > > > > > > > > > >> > > > > > > > > And just in case.... > > >> > > > > > > > > > > >> > > > > > > > > swagger: '2.0' > > >> > > > > > > > > info: > > >> > > > > > > > > version: 0.0.1 > > >> > > > > > > > > title: Myriad API > > >> > > > > > > > > description: | > > >> > > > > > > > > Myriad API description > > >> > > > > > > > > basePath: /api > > >> > > > > > > > > schemes: > > >> > > > > > > > > - http > > >> > > > > > > > > consumes: > > >> > > > > > > > > - application/json > > >> > > > > > > > > - text/plain > > >> > > > > > > > > produces: > > >> > > > > > > > > - application/json > > >> > > > > > > > > - text/plain > > >> > > > > > > > > paths: > > >> > > > > > > > > /cluster/flexup: > > >> > > > > > > > > put: > > >> > > > > > > > > parameters: > > >> > > > > > > > > - name: instance > > >> > > > > > > > > in: body > > >> > > > > > > > > description: Instance profile and quantity to > > >> flex > > >> > up > > >> > > > > > > > > schema: > > >> > > > > > > > > $ref: '#/definitions/FlexUp' > > >> > > > > > > > > required: true > > >> > > > > > > > > description: Original Flexup API > > >> > > > > > > > > responses: > > >> > > > > > > > > 200: > > >> > > > > > > > > description: OK > > >> > > > > > > > > /cluster/flexup/profile/{profile-name}: > > >> > > > > > > > > put: > > >> > > > > > > > > parameters: > > >> > > > > > > > > - name: profile-name > > >> > > > > > > > > in: path > > >> > > > > > > > > type: string > > >> > > > > > > > > description: Instance profile name > > >> > > > > > > > > description: Flexup a single instance of a > > specified > > >> > > > profile > > >> > > > > > > > > responses: > > >> > > > > > > > > 200: > > >> > > > > > > > > description: OK > > >> > > > > > > > > /cluster/flexup/profile/{profile-name}/{instances}: > > >> > > > > > > > > put: > > >> > > > > > > > > parameters: > > >> > > > > > > > > - name: profile-name > > >> > > > > > > > > in: path > > >> > > > > > > > > type: string > > >> > > > > > > > > description: Instance profile name > > >> > > > > > > > > - name: instances > > >> > > > > > > > > in: path > > >> > > > > > > > > type: integer > > >> > > > > > > > > description: Number of instances > > >> > > > > > > > > description: Flexup a multiple instances of a > > >> specified > > >> > > > > profile > > >> > > > > > > > > responses: > > >> > > > > > > > > 200: > > >> > > > > > > > > description: OK > > >> > > > > > > > > /cluster/flexdown: > > >> > > > > > > > > put: > > >> > > > > > > > > parameters: > > >> > > > > > > > > - name: instance > > >> > > > > > > > > in: body > > >> > > > > > > > > description: Number of instances to flex down > > >> > > > > > > > > schema: > > >> > > > > > > > > $ref: '#/definitions/FlexDown' > > >> > > > > > > > > required: true > > >> > > > > > > > > description: Original Flexdown API > > >> > > > > > > > > responses: > > >> > > > > > > > > 200: > > >> > > > > > > > > description: OK > > >> > > > > > > > > /cluster/flexdown/profile/{profile-name}: > > >> > > > > > > > > put: > > >> > > > > > > > > parameters: > > >> > > > > > > > > - name: profile-name > > >> > > > > > > > > in: path > > >> > > > > > > > > type: string > > >> > > > > > > > > description: Instance profile name > > >> > > > > > > > > description: Flexup a single instance of a > > specified > > >> > > > profile > > >> > > > > > > > > responses: > > >> > > > > > > > > 200: > > >> > > > > > > > > description: OK > > >> > > > > > > > > /cluster/flexdown/profile/{profile-name}/{instances}: > > >> > > > > > > > > put: > > >> > > > > > > > > parameters: > > >> > > > > > > > > - name: profile-name > > >> > > > > > > > > in: path > > >> > > > > > > > > type: string > > >> > > > > > > > > description: Instance profile name > > >> > > > > > > > > - name: instances > > >> > > > > > > > > in: path > > >> > > > > > > > > type: integer > > >> > > > > > > > > description: Number of instances > > >> > > > > > > > > description: Flexdown a multiple instances of a > > >> > specified > > >> > > > > > profile > > >> > > > > > > > > responses: > > >> > > > > > > > > 200: > > >> > > > > > > > > description: OK > > >> > > > > > > > > /cluster/flexdown/instance/{instance-id}: > > >> > > > > > > > > put: > > >> > > > > > > > > parameters: > > >> > > > > > > > > - name: instance-id > > >> > > > > > > > > in: path > > >> > > > > > > > > type: string > > >> > > > > > > > > description: Instance profile name > > >> > > > > > > > > description: Flexup a single instance of a > > specified > > >> > > > profile > > >> > > > > > > > > responses: > > >> > > > > > > > > 200: > > >> > > > > > > > > description: OK > > >> > > > > > > > > definitions: > > >> > > > > > > > > FlexUp: > > >> > > > > > > > > properties: > > >> > > > > > > > > profile: > > >> > > > > > > > > type: string > > >> > > > > > > > > instances: > > >> > > > > > > > > type: integer > > >> > > > > > > > > format: int32 > > >> > > > > > > > > FlexDown: > > >> > > > > > > > > properties: > > >> > > > > > > > > instances: > > >> > > > > > > > > type: integer > > >> > > > > > > > > format: int32 > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella < > > >> > > > > > > [email protected]> > > >> > > > > > > > > wrote: > > >> > > > > > > > > > > >> > > > > > > > > > Hi Jim, > > >> > > > > > > > > > > > >> > > > > > > > > > Did you attach a file? I think the apache mailing > > list > > >> > > > > swallowed > > >> > > > > > > the > > >> > > > > > > > > > attachment. Can you please send us the document > > contents > > >> in > > >> > > > plain > > >> > > > > > > text? > > >> > > > > > > > > > > > >> > > > > > > > > > Thanks, > > >> > > > > > > > > > Santosh > > >> > > > > > > > > > > > >> > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar < > > >> > > [email protected]> > > >> > > > > > > wrote: > > >> > > > > > > > > > > > >> > > > > > > > > > > This is in regards to > > >> > > > > https://github.com/mesos/myriad/issues/89 > > >> > > > > > > > > > > > > >> > > > > > > > > > > I created a swagger document (attached) of the > > current > > >> > flex > > >> > > > > APIs > > >> > > > > > > and > > >> > > > > > > > > what > > >> > > > > > > > > > > I propose for the new flex apis. If people like > > >> swagger, > > >> > I > > >> > > > can > > >> > > > > > > > document > > >> > > > > > > > > > the > > >> > > > > > > > > > > rest of the API, including the correct response > > codes. > > >> > > Either > > >> > > > > > way I > > >> > > > > > > > > think > > >> > > > > > > > > > > we should discuss what the API should be going > > >> forward. > > >> > > > > > > > > > > > > >> > > > > > > > > > > Paste the file contents into > > >> http://editor.swagger.io/ > > >> > and > > >> > > > > > you'll > > >> > > > > > > > get > > >> > > > > > > > > > > nice HTML to browse. > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > > > > > >
