> One additional thing I'd like is the ability to flexup a nmnode on all > agents with a certain attribute as opposed to a fixed number.
Certainly! Marathon places tasks on agents sharing a common attribute. Myriad could/should do the same for placing NMs. If each agent running HDFS data node shares an attribute such as "dfs", then NMs could always be launched on the "dfs" nodes. This improves data locality for YARN jobs dramatically. Santosh On Thu, Aug 27, 2015 at 2:47 PM, Darin Johnson <[email protected]> wrote: > One additional thing I'd like is the ability to flexup a nmnode on all > agents with a certain attribute as opposed to a fixed number. > > > Also, when I get back from vacation I plan on scoping mesos-1739 (dynamic > attributes) which would allow for tighter integration with the hdfs > framework. An alternative would be to get hostnames from the namenode, > though not as seemless. > (Reviving this thread) > > We've discussed several great points in the thread (PUT vs POST, need for > GET, JSON payload vs parameters in URL, declarative interface etc). > Just to get us going, I think we should focus on a couple of things that > will be useful for Myriad users, while leaving them flexible enough to be > evolved in the future. > > What I heard from several folks (some of it brought up again at MesosCon) > about the flex up/down APIs is this: > - flexup doesn't support launching NMs on specific set of hosts. This is > especially needed to launch NMs on same set of nodes that have HDFS > DataNode running. > - flexdown lacks an option to shut down NMs with a specific profile. Today, > we bring down ANY arbitrary NM. > - flexdown lacks an option to shutdown NMs running on specific hosts. > > I captured my thoughts in a document here: > > https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t > > @Jim: Happy to collaborate at one single place (Swagger/Google Doc) to > finalize the APIs. Just let me know. > > Thanks, > Santosh > > On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <[email protected]> wrote: > > > Seems like POST is a winner with people. > > > > Another thing to consider is how we want the REST interface to be vs what > > we want the UI to do. The UI could support flexup/flexdown like it is > while > > the REST interface is just a declarative state like Adam suggested. The > UI > > would just be responsible for translating the request into the new state. > > > > Tomorrow I'll try to put together another swagger doc with some of the > > suggested options. > > > > > > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman > > <[email protected] > > > wrote: > > > > > I think we are at the point to list all the options we want "flex" API > to > > > support. > > > 1. Do we continue supporting flexup/down or just "flex" with additional > > > "preposition" like up/down:https://hostname:port/flex/up(down) > > > 2. I think we should switch to POST and may be maintain PUT for legacy > > (if > > > even needed to keep it). We are not DB after all and not storing any > > > retrievable info here :) > > > 3. We need to add status (GET) to see the status - though I think we > have > > > one > > > 4. Define JSON payload to support different cases a. providing > > > different profiles together: [{profile:"big", > > > instances:2},{profile:"medium",instances:6}] b. provide what state > we > > > want Myriad to be in: "I want 10 medium instances" and then Myriad will > > do > > > whatever isnecessary to transition to that state, > > adding/removing/resizing > > > NMs" c. flex/down particular instance IDs d. flex up/down > preferred > > > hosts, delays, others > > > 5. How all this fits into FineGrain Scaling? With it we would do > > automatic > > > flex up/down. And the less knobs admin will have to turn the easier it > is > > > for admin and the end users. > > > > > > From: Adam Bordelon <[email protected]> > > > To: [email protected] > > > Sent: Sunday, June 14, 2015 2:54 PM > > > Subject: Re: Flex API > > > > > > (In addition,) I'd also like to see a more declarative interface. > Instead > > > of "add two more instances", the user(s) could just specify the desired > > > state of "I want 10 medium instances" and then Myriad will do whatever > is > > > necessary to transition to that state, adding/removing/resizing NMs as > > > necessary. > > > > > > > > > > > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena < > > > [email protected] > > > > wrote: > > > > > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <[email protected]> > wrote: > > > > > > > > > What verb to use when outside of database land can be argued. I > would > > > > vote > > > > > for POST over PUT just because I tend to default to POST. PUT was > > there > > > > > when I showed up, so I left it. > > > > > > > > > > > > Last time I agonized about PUT vs POST the most logical distinction I > > > found > > > > was that PUT should be used for idempotent operations, while POST for > > > > non-idempotent (like we have here with flex-up, since instance-ids > are > > > > generated). > > > > > > > > Since the api doesn't wait until the > > > > > instances are created to return, we can't really return the > instance > > > IDs > > > > we > > > > > created. > > > > > > > > > > > > > That seems OK to me. > > > > > > > > > > > > > The GET would just return some status? > > > > > > > > > > > > > Yeah, I was thinking that this would be needed for a future GUI where > > we > > > > list all instances with parameters and status for each (profile, > > current > > > > cpu/ram/disk, node, uptime). I'm picturing checkboxes next to each > so > > > > users can multi-select and hit 'delete' to wipe them away (like > > flex-down > > > > does now). > > > > > > > > The PATCH is interesting > > > > > > > > > > > > > Yeah, I started to write PUT but to REST geeks PUT implies you always > > > have > > > > to rewrite the complete object when making changes. PATCH allows > more > > > > flexible modifications. > > > > > > > > The DELETE makes sense to me. > > > > > > > > > > Your use of instances vs instance is interesting. Perhaps we want > to > > > > > support POST [{profile:"big", instances:2},{profile:"medium", > > > > instances:6}] > > > > > > > > > > > > > Yeah, that'd be cool! > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena < > > > > > [email protected] > > > > > > wrote: > > > > > > > > > > > Any reason we need to make it action-based (flex-up/flex-down) > > rather > > > > > than > > > > > > instance outcome-based? The way i've seen similar things modeled > > in > > > > > other > > > > > > REST APIs is similar to below. This may also fit better into > REST > > > > client > > > > > > frameworks. > > > > > > > > > > > > Create - POST /instances/ {"profile" : "zero", "instances" : 2, > > > > "delay" : > > > > > > 2, "preferredHosts" : ["host1", "host2"]} > > > > > > Read - GET /instances/ or GET /instances/<instanceid> > > > > > > Update (resize?) - PATCH /instances/<instanceid> {"profile" : > > "big"} > > > > > > Delete - DELETE /instances/<instanceid> > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar <[email protected]> > > > wrote: > > > > > > > > > > > > > That is always an issue with parameters in the URL, and there's > > > > nothing > > > > > > > worse than parameters in the URL and having to put JSON in the > > body > > > > > > because > > > > > > > we want to extend it later. Currently flexup takes a JSON blob > in > > > the > > > > > > body > > > > > > > which we could add more to like you showed. I guess the > question > > is > > > > if > > > > > we > > > > > > > want to support a quick and dirty flexup api via URL > parameters. > > I > > > > > > wouldn't > > > > > > > go any further than these parameters, but all this would do at > > this > > > > > point > > > > > > > is make testing with a curl command easier. Getting rid of it > for > > > > > flexup > > > > > > is > > > > > > > fine with me. > > > > > > > > > > > > > > I do like the flexdown/instance/{instance-id} though. We should > > > > > probably > > > > > > do > > > > > > > one with flexdown/instance with a json array of instance-ids > too > > > > > though. > > > > > > > > > > > > > > Also with the parameter verification patch I just submitted, if > > you > > > > ask > > > > > > to > > > > > > > flexdown 10 instances and you only have 5 running it will just > > log > > > a > > > > > > > warning and kill all 5. Not sure if that is the desired > behavior. > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella < > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > I think rather than having an API defined as > > > > > > > > "/cluster/flexup/profile/{profile-name}", it's better to > define > > > it > > > > > just > > > > > > > as > > > > > > > > "/cluster/flexup" and have parameters such as "profile: > > > > > > > > <a-value-for-profile-name>" in the JSON payload. The reason > is, > > > if > > > > > the > > > > > > > > params are added into the API endpoint, it becomes less > > flexible > > > to > > > > > > > evolve. > > > > > > > > For e.g. /cluster/flexup currently can take just "profile" > and > > > > > > > "instances" > > > > > > > > in it's payload, but in the future can optionally take > > parameters > > > > > such > > > > > > as > > > > > > > > "preferredHosts: [host1, host2, host9]", "delay: > > > > > > > > <take_some_sweet_time_before_flexing_up>" etc. > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > Santosh > > > > > > > > > > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > I thought that might happen. I created a gist here: > > > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e > > > > > > > > > > > > > > > > > > And just in case.... > > > > > > > > > > > > > > > > > > swagger: '2.0' > > > > > > > > > info: > > > > > > > > > version: 0.0.1 > > > > > > > > > title: Myriad API > > > > > > > > > description: | > > > > > > > > > Myriad API description > > > > > > > > > basePath: /api > > > > > > > > > schemes: > > > > > > > > > - http > > > > > > > > > consumes: > > > > > > > > > - application/json > > > > > > > > > - text/plain > > > > > > > > > produces: > > > > > > > > > - application/json > > > > > > > > > - text/plain > > > > > > > > > paths: > > > > > > > > > /cluster/flexup: > > > > > > > > > put: > > > > > > > > > parameters: > > > > > > > > > - name: instance > > > > > > > > > in: body > > > > > > > > > description: Instance profile and quantity to flex > > up > > > > > > > > > schema: > > > > > > > > > $ref: '#/definitions/FlexUp' > > > > > > > > > required: true > > > > > > > > > description: Original Flexup API > > > > > > > > > responses: > > > > > > > > > 200: > > > > > > > > > description: OK > > > > > > > > > /cluster/flexup/profile/{profile-name}: > > > > > > > > > put: > > > > > > > > > parameters: > > > > > > > > > - name: profile-name > > > > > > > > > in: path > > > > > > > > > type: string > > > > > > > > > description: Instance profile name > > > > > > > > > description: Flexup a single instance of a specified > > > > profile > > > > > > > > > responses: > > > > > > > > > 200: > > > > > > > > > description: OK > > > > > > > > > /cluster/flexup/profile/{profile-name}/{instances}: > > > > > > > > > put: > > > > > > > > > parameters: > > > > > > > > > - name: profile-name > > > > > > > > > in: path > > > > > > > > > type: string > > > > > > > > > description: Instance profile name > > > > > > > > > - name: instances > > > > > > > > > in: path > > > > > > > > > type: integer > > > > > > > > > description: Number of instances > > > > > > > > > description: Flexup a multiple instances of a > specified > > > > > profile > > > > > > > > > responses: > > > > > > > > > 200: > > > > > > > > > description: OK > > > > > > > > > /cluster/flexdown: > > > > > > > > > put: > > > > > > > > > parameters: > > > > > > > > > - name: instance > > > > > > > > > in: body > > > > > > > > > description: Number of instances to flex down > > > > > > > > > schema: > > > > > > > > > $ref: '#/definitions/FlexDown' > > > > > > > > > required: true > > > > > > > > > description: Original Flexdown API > > > > > > > > > responses: > > > > > > > > > 200: > > > > > > > > > description: OK > > > > > > > > > /cluster/flexdown/profile/{profile-name}: > > > > > > > > > put: > > > > > > > > > parameters: > > > > > > > > > - name: profile-name > > > > > > > > > in: path > > > > > > > > > type: string > > > > > > > > > description: Instance profile name > > > > > > > > > description: Flexup a single instance of a specified > > > > profile > > > > > > > > > responses: > > > > > > > > > 200: > > > > > > > > > description: OK > > > > > > > > > /cluster/flexdown/profile/{profile-name}/{instances}: > > > > > > > > > put: > > > > > > > > > parameters: > > > > > > > > > - name: profile-name > > > > > > > > > in: path > > > > > > > > > type: string > > > > > > > > > description: Instance profile name > > > > > > > > > - name: instances > > > > > > > > > in: path > > > > > > > > > type: integer > > > > > > > > > description: Number of instances > > > > > > > > > description: Flexdown a multiple instances of a > > specified > > > > > > profile > > > > > > > > > responses: > > > > > > > > > 200: > > > > > > > > > description: OK > > > > > > > > > /cluster/flexdown/instance/{instance-id}: > > > > > > > > > put: > > > > > > > > > parameters: > > > > > > > > > - name: instance-id > > > > > > > > > in: path > > > > > > > > > type: string > > > > > > > > > description: Instance profile name > > > > > > > > > description: Flexup a single instance of a specified > > > > profile > > > > > > > > > responses: > > > > > > > > > 200: > > > > > > > > > description: OK > > > > > > > > > definitions: > > > > > > > > > FlexUp: > > > > > > > > > properties: > > > > > > > > > profile: > > > > > > > > > type: string > > > > > > > > > instances: > > > > > > > > > type: integer > > > > > > > > > format: int32 > > > > > > > > > FlexDown: > > > > > > > > > properties: > > > > > > > > > instances: > > > > > > > > > type: integer > > > > > > > > > format: int32 > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella < > > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Jim, > > > > > > > > > > > > > > > > > > > > Did you attach a file? I think the apache mailing list > > > > > swallowed > > > > > > > the > > > > > > > > > > attachment. Can you please send us the document contents > in > > > > plain > > > > > > > text? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Santosh > > > > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > This is in regards to > > > > > https://github.com/mesos/myriad/issues/89 > > > > > > > > > > > > > > > > > > > > > > I created a swagger document (attached) of the current > > flex > > > > > APIs > > > > > > > and > > > > > > > > > what > > > > > > > > > > > I propose for the new flex apis. If people like > swagger, > > I > > > > can > > > > > > > > document > > > > > > > > > > the > > > > > > > > > > > rest of the API, including the correct response codes. > > > Either > > > > > > way I > > > > > > > > > think > > > > > > > > > > > we should discuss what the API should be going forward. > > > > > > > > > > > > > > > > > > > > > > Paste the file contents into http://editor.swagger.io/ > > and > > > > > > you'll > > > > > > > > get > > > > > > > > > > > nice HTML to browse. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
