Santosh, I can do the UI work. Should I make it a separate PR or want me to push you a patch on the side?
On Wed, Sep 23, 2015 at 6:58 PM, Santosh Marella <[email protected]> wrote: > Addressed review comments from Yuliya and updated the PR: > https://github.com/mesos/myriad/pull/129 > > @Jim - Your UI expertise is much sought to support "constraints" parameter > in both flexup/down and "profile" in flexdown. > The API doc is updated as part of the PR: > > https://github.com/mesos/myriad/pull/129/files#diff-96cc378dcdef180641176501f42183c7 > > Thanks, > Santosh > > On Mon, Sep 14, 2015 at 10:54 AM, Santosh Marella <[email protected]> > wrote: > > > The proposal is to have "constraints" parameter supported for both > > flexup/flexdown. > > So, one could flexdown NMs on specific hosts, for e..g by providing a > > constraint > > around "hostname" (/api/flexdown {"constraints": [["hostname" LIKE " > > specific.host.com"]]}. > > > > Santosh > > > > On Mon, Sep 14, 2015 at 2:31 AM, Adam Bordelon <[email protected]> > wrote: > > > >> +1, but EQUALS might be an easier starting point than LIKE. Your choice. > >> What about flexing down specific hosts? Is that out of scope for now? > >> > >> On Wed, Sep 2, 2015 at 2:13 PM, Santosh Marella <[email protected]> > >> wrote: > >> > >> > Ping! Checking if there is anymore feedback on the APIs for > flexup/down. > >> > > >> > Here is a textual summary of what's being proposed: > >> > 1. Add a new “constraints” parameter in both flexup and flexdown APIs > >> (just > >> > like how Marathon does). > >> > Initially, start with support for LIKE operator in "constraints" that > >> > operates on “hostname” and “mesos slave attribute > >> > <http://mesos.apache.org/documentation/attributes-resources/>” fields > >> > using > >> > regex. > >> > 2. Add a new “profile” parameter in the flexdown API. > >> > > >> > #1 would give admins a better control over where the NMs should be > >> launched > >> > or which specific NMs should be shut down. > >> > #2 helps with shutting down multiple NMs of same "profile" in one > shot. > >> > > >> > If the above sounds reasonable to start with, I'd like to start > working > >> on > >> > the code changes. > >> > > >> > Thanks, > >> > Santosh > >> > > >> > On Thu, Aug 27, 2015 at 4:25 PM, Santosh Marella < > [email protected] > >> > > >> > wrote: > >> > > >> > > > One additional thing I'd like is the ability to flexup a nmnode on > >> all > >> > > > agents with a certain attribute as opposed to a fixed number. > >> > > > >> > > Certainly! Marathon places tasks on agents sharing a common > attribute. > >> > > Myriad could/should do the same for placing NMs. If each agent > running > >> > > HDFS data node shares an attribute such as "dfs", then NMs could > >> always > >> > > be launched on the "dfs" nodes. This improves data locality for YARN > >> jobs > >> > > dramatically. > >> > > > >> > > Santosh > >> > > > >> > > On Thu, Aug 27, 2015 at 2:47 PM, Darin Johnson < > >> [email protected]> > >> > > wrote: > >> > > > >> > >> One additional thing I'd like is the ability to flexup a nmnode on > >> all > >> > >> agents with a certain attribute as opposed to a fixed number. > >> > >> > >> > > > >> > > > >> > >> > >> > >> Also, when I get back from vacation I plan on scoping mesos-1739 > >> > (dynamic > >> > >> attributes) which would allow for tighter integration with the hdfs > >> > >> framework. An alternative would be to get hostnames from the > >> namenode, > >> > >> though not as seemless. > >> > >> (Reviving this thread) > >> > >> > >> > >> We've discussed several great points in the thread (PUT vs POST, > need > >> > for > >> > >> GET, JSON payload vs parameters in URL, declarative interface etc). > >> > >> Just to get us going, I think we should focus on a couple of things > >> that > >> > >> will be useful for Myriad users, while leaving them flexible enough > >> to > >> > be > >> > >> evolved in the future. > >> > >> > >> > >> What I heard from several folks (some of it brought up again at > >> > MesosCon) > >> > >> about the flex up/down APIs is this: > >> > >> - flexup doesn't support launching NMs on specific set of hosts. > >> This is > >> > >> especially needed to launch NMs on same set of nodes that have HDFS > >> > >> DataNode running. > >> > >> - flexdown lacks an option to shut down NMs with a specific > profile. > >> > >> Today, > >> > >> we bring down ANY arbitrary NM. > >> > >> - flexdown lacks an option to shutdown NMs running on specific > hosts. > >> > >> > >> > >> I captured my thoughts in a document here: > >> > >> > >> > >> > >> > > >> > https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t > >> > >> > >> > >> @Jim: Happy to collaborate at one single place (Swagger/Google Doc) > >> to > >> > >> finalize the APIs. Just let me know. > >> > >> > >> > >> Thanks, > >> > >> Santosh > >> > >> > >> > >> On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <[email protected]> > >> wrote: > >> > >> > >> > >> > Seems like POST is a winner with people. > >> > >> > > >> > >> > Another thing to consider is how we want the REST interface to be > >> vs > >> > >> what > >> > >> > we want the UI to do. The UI could support flexup/flexdown like > it > >> is > >> > >> while > >> > >> > the REST interface is just a declarative state like Adam > suggested. > >> > The > >> > >> UI > >> > >> > would just be responsible for translating the request into the > new > >> > >> state. > >> > >> > > >> > >> > Tomorrow I'll try to put together another swagger doc with some > of > >> the > >> > >> > suggested options. > >> > >> > > >> > >> > > >> > >> > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman > >> > >> > <[email protected] > >> > >> > > wrote: > >> > >> > > >> > >> > > I think we are at the point to list all the options we want > >> "flex" > >> > API > >> > >> to > >> > >> > > support. > >> > >> > > 1. Do we continue supporting flexup/down or just "flex" with > >> > >> additional > >> > >> > > "preposition" like up/down:https://hostname:port/flex/up(down) > >> > >> > > 2. I think we should switch to POST and may be maintain PUT for > >> > legacy > >> > >> > (if > >> > >> > > even needed to keep it). We are not DB after all and not > storing > >> any > >> > >> > > retrievable info here :) > >> > >> > > 3. We need to add status (GET) to see the status - though I > >> think we > >> > >> have > >> > >> > > one > >> > >> > > 4. Define JSON payload to support different cases a. > providing > >> > >> > > different profiles together: [{profile:"big", > >> > >> > > instances:2},{profile:"medium",instances:6}] b. provide what > >> > state > >> > >> we > >> > >> > > want Myriad to be in: "I want 10 medium instances" and then > >> Myriad > >> > >> will > >> > >> > do > >> > >> > > whatever isnecessary to transition to that state, > >> > >> > adding/removing/resizing > >> > >> > > NMs" c. flex/down particular instance IDs d. flex up/down > >> > >> preferred > >> > >> > > hosts, delays, others > >> > >> > > 5. How all this fits into FineGrain Scaling? With it we would > do > >> > >> > automatic > >> > >> > > flex up/down. And the less knobs admin will have to turn the > >> easier > >> > it > >> > >> is > >> > >> > > for admin and the end users. > >> > >> > > > >> > >> > > From: Adam Bordelon <[email protected]> > >> > >> > > To: [email protected] > >> > >> > > Sent: Sunday, June 14, 2015 2:54 PM > >> > >> > > Subject: Re: Flex API > >> > >> > > > >> > >> > > (In addition,) I'd also like to see a more declarative > interface. > >> > >> Instead > >> > >> > > of "add two more instances", the user(s) could just specify the > >> > >> desired > >> > >> > > state of "I want 10 medium instances" and then Myriad will do > >> > whatever > >> > >> is > >> > >> > > necessary to transition to that state, adding/removing/resizing > >> NMs > >> > as > >> > >> > > necessary. > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena < > >> > >> > > [email protected] > >> > >> > > > wrote: > >> > >> > > > >> > >> > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar < > [email protected]> > >> > >> wrote: > >> > >> > > > > >> > >> > > > > What verb to use when outside of database land can be > >> argued. I > >> > >> would > >> > >> > > > vote > >> > >> > > > > for POST over PUT just because I tend to default to POST. > PUT > >> > was > >> > >> > there > >> > >> > > > > when I showed up, so I left it. > >> > >> > > > > >> > >> > > > > >> > >> > > > Last time I agonized about PUT vs POST the most logical > >> > distinction > >> > >> I > >> > >> > > found > >> > >> > > > was that PUT should be used for idempotent operations, while > >> POST > >> > >> for > >> > >> > > > non-idempotent (like we have here with flex-up, since > >> instance-ids > >> > >> are > >> > >> > > > generated). > >> > >> > > > > >> > >> > > > Since the api doesn't wait until the > >> > >> > > > > instances are created to return, we can't really return the > >> > >> instance > >> > >> > > IDs > >> > >> > > > we > >> > >> > > > > created. > >> > >> > > > > > >> > >> > > > > >> > >> > > > That seems OK to me. > >> > >> > > > > >> > >> > > > > >> > >> > > > > The GET would just return some status? > >> > >> > > > > > >> > >> > > > > >> > >> > > > Yeah, I was thinking that this would be needed for a future > GUI > >> > >> where > >> > >> > we > >> > >> > > > list all instances with parameters and status for each > >> (profile, > >> > >> > current > >> > >> > > > cpu/ram/disk, node, uptime). I'm picturing checkboxes next > to > >> > each > >> > >> so > >> > >> > > > users can multi-select and hit 'delete' to wipe them away > (like > >> > >> > flex-down > >> > >> > > > does now). > >> > >> > > > > >> > >> > > > The PATCH is interesting > >> > >> > > > > > >> > >> > > > > >> > >> > > > Yeah, I started to write PUT but to REST geeks PUT implies > you > >> > >> always > >> > >> > > have > >> > >> > > > to rewrite the complete object when making changes. PATCH > >> allows > >> > >> more > >> > >> > > > flexible modifications. > >> > >> > > > > >> > >> > > > The DELETE makes sense to me. > >> > >> > > > > > >> > >> > > > > Your use of instances vs instance is interesting. Perhaps > we > >> > want > >> > >> to > >> > >> > > > > support POST [{profile:"big", > instances:2},{profile:"medium", > >> > >> > > > instances:6}] > >> > >> > > > > > >> > >> > > > > >> > >> > > > Yeah, that'd be cool! > >> > >> > > > > >> > >> > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena < > >> > >> > > > > [email protected] > >> > >> > > > > > wrote: > >> > >> > > > > > >> > >> > > > > > Any reason we need to make it action-based > >> (flex-up/flex-down) > >> > >> > rather > >> > >> > > > > than > >> > >> > > > > > instance outcome-based? The way i've seen similar things > >> > >> modeled > >> > >> > in > >> > >> > > > > other > >> > >> > > > > > REST APIs is similar to below. This may also fit better > >> into > >> > >> REST > >> > >> > > > client > >> > >> > > > > > frameworks. > >> > >> > > > > > > >> > >> > > > > > Create - POST /instances/ {"profile" : "zero", > "instances" > >> : > >> > 2, > >> > >> > > > "delay" : > >> > >> > > > > > 2, "preferredHosts" : ["host1", "host2"]} > >> > >> > > > > > Read - GET /instances/ or GET /instances/<instanceid> > >> > >> > > > > > Update (resize?) - PATCH /instances/<instanceid> > >> {"profile" : > >> > >> > "big"} > >> > >> > > > > > Delete - DELETE /instances/<instanceid> > >> > >> > > > > > > >> > >> > > > > > Thoughts? > >> > >> > > > > > > >> > >> > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar < > >> [email protected] > >> > > > >> > >> > > wrote: > >> > >> > > > > > > >> > >> > > > > > > That is always an issue with parameters in the URL, and > >> > >> there's > >> > >> > > > nothing > >> > >> > > > > > > worse than parameters in the URL and having to put JSON > >> in > >> > the > >> > >> > body > >> > >> > > > > > because > >> > >> > > > > > > we want to extend it later. Currently flexup takes a > JSON > >> > blob > >> > >> in > >> > >> > > the > >> > >> > > > > > body > >> > >> > > > > > > which we could add more to like you showed. I guess the > >> > >> question > >> > >> > is > >> > >> > > > if > >> > >> > > > > we > >> > >> > > > > > > want to support a quick and dirty flexup api via URL > >> > >> parameters. > >> > >> > I > >> > >> > > > > > wouldn't > >> > >> > > > > > > go any further than these parameters, but all this > would > >> do > >> > at > >> > >> > this > >> > >> > > > > point > >> > >> > > > > > > is make testing with a curl command easier. Getting rid > >> of > >> > it > >> > >> for > >> > >> > > > > flexup > >> > >> > > > > > is > >> > >> > > > > > > fine with me. > >> > >> > > > > > > > >> > >> > > > > > > I do like the flexdown/instance/{instance-id} though. > We > >> > >> should > >> > >> > > > > probably > >> > >> > > > > > do > >> > >> > > > > > > one with flexdown/instance with a json array of > >> instance-ids > >> > >> too > >> > >> > > > > though. > >> > >> > > > > > > > >> > >> > > > > > > Also with the parameter verification patch I just > >> submitted, > >> > >> if > >> > >> > you > >> > >> > > > ask > >> > >> > > > > > to > >> > >> > > > > > > flexdown 10 instances and you only have 5 running it > will > >> > just > >> > >> > log > >> > >> > > a > >> > >> > > > > > > warning and kill all 5. Not sure if that is the desired > >> > >> behavior. > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella < > >> > >> > > > > [email protected]> > >> > >> > > > > > > wrote: > >> > >> > > > > > > > >> > >> > > > > > > > I think rather than having an API defined as > >> > >> > > > > > > > "/cluster/flexup/profile/{profile-name}", it's better > >> to > >> > >> define > >> > >> > > it > >> > >> > > > > just > >> > >> > > > > > > as > >> > >> > > > > > > > "/cluster/flexup" and have parameters such as > "profile: > >> > >> > > > > > > > <a-value-for-profile-name>" in the JSON payload. The > >> > reason > >> > >> is, > >> > >> > > if > >> > >> > > > > the > >> > >> > > > > > > > params are added into the API endpoint, it becomes > less > >> > >> > flexible > >> > >> > > to > >> > >> > > > > > > evolve. > >> > >> > > > > > > > For e.g. /cluster/flexup currently can take just > >> "profile" > >> > >> and > >> > >> > > > > > > "instances" > >> > >> > > > > > > > in it's payload, but in the future can optionally > take > >> > >> > parameters > >> > >> > > > > such > >> > >> > > > > > as > >> > >> > > > > > > > "preferredHosts: [host1, host2, host9]", "delay: > >> > >> > > > > > > > <take_some_sweet_time_before_flexing_up>" etc. > >> > >> > > > > > > > > >> > >> > > > > > > > Thoughts? > >> > >> > > > > > > > > >> > >> > > > > > > > Santosh > >> > >> > > > > > > > > >> > >> > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar < > >> > >> [email protected]> > >> > >> > > > > wrote: > >> > >> > > > > > > > > >> > >> > > > > > > > > I thought that might happen. I created a gist here: > >> > >> > > > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e > >> > >> > > > > > > > > > >> > >> > > > > > > > > And just in case.... > >> > >> > > > > > > > > > >> > >> > > > > > > > > swagger: '2.0' > >> > >> > > > > > > > > info: > >> > >> > > > > > > > > version: 0.0.1 > >> > >> > > > > > > > > title: Myriad API > >> > >> > > > > > > > > description: | > >> > >> > > > > > > > > Myriad API description > >> > >> > > > > > > > > basePath: /api > >> > >> > > > > > > > > schemes: > >> > >> > > > > > > > > - http > >> > >> > > > > > > > > consumes: > >> > >> > > > > > > > > - application/json > >> > >> > > > > > > > > - text/plain > >> > >> > > > > > > > > produces: > >> > >> > > > > > > > > - application/json > >> > >> > > > > > > > > - text/plain > >> > >> > > > > > > > > paths: > >> > >> > > > > > > > > /cluster/flexup: > >> > >> > > > > > > > > put: > >> > >> > > > > > > > > parameters: > >> > >> > > > > > > > > - name: instance > >> > >> > > > > > > > > in: body > >> > >> > > > > > > > > description: Instance profile and quantity > >> to > >> > >> flex > >> > >> > up > >> > >> > > > > > > > > schema: > >> > >> > > > > > > > > $ref: '#/definitions/FlexUp' > >> > >> > > > > > > > > required: true > >> > >> > > > > > > > > description: Original Flexup API > >> > >> > > > > > > > > responses: > >> > >> > > > > > > > > 200: > >> > >> > > > > > > > > description: OK > >> > >> > > > > > > > > /cluster/flexup/profile/{profile-name}: > >> > >> > > > > > > > > put: > >> > >> > > > > > > > > parameters: > >> > >> > > > > > > > > - name: profile-name > >> > >> > > > > > > > > in: path > >> > >> > > > > > > > > type: string > >> > >> > > > > > > > > description: Instance profile name > >> > >> > > > > > > > > description: Flexup a single instance of a > >> > specified > >> > >> > > > profile > >> > >> > > > > > > > > responses: > >> > >> > > > > > > > > 200: > >> > >> > > > > > > > > description: OK > >> > >> > > > > > > > > > /cluster/flexup/profile/{profile-name}/{instances}: > >> > >> > > > > > > > > put: > >> > >> > > > > > > > > parameters: > >> > >> > > > > > > > > - name: profile-name > >> > >> > > > > > > > > in: path > >> > >> > > > > > > > > type: string > >> > >> > > > > > > > > description: Instance profile name > >> > >> > > > > > > > > - name: instances > >> > >> > > > > > > > > in: path > >> > >> > > > > > > > > type: integer > >> > >> > > > > > > > > description: Number of instances > >> > >> > > > > > > > > description: Flexup a multiple instances of a > >> > >> specified > >> > >> > > > > profile > >> > >> > > > > > > > > responses: > >> > >> > > > > > > > > 200: > >> > >> > > > > > > > > description: OK > >> > >> > > > > > > > > /cluster/flexdown: > >> > >> > > > > > > > > put: > >> > >> > > > > > > > > parameters: > >> > >> > > > > > > > > - name: instance > >> > >> > > > > > > > > in: body > >> > >> > > > > > > > > description: Number of instances to flex > >> down > >> > >> > > > > > > > > schema: > >> > >> > > > > > > > > $ref: '#/definitions/FlexDown' > >> > >> > > > > > > > > required: true > >> > >> > > > > > > > > description: Original Flexdown API > >> > >> > > > > > > > > responses: > >> > >> > > > > > > > > 200: > >> > >> > > > > > > > > description: OK > >> > >> > > > > > > > > /cluster/flexdown/profile/{profile-name}: > >> > >> > > > > > > > > put: > >> > >> > > > > > > > > parameters: > >> > >> > > > > > > > > - name: profile-name > >> > >> > > > > > > > > in: path > >> > >> > > > > > > > > type: string > >> > >> > > > > > > > > description: Instance profile name > >> > >> > > > > > > > > description: Flexup a single instance of a > >> > specified > >> > >> > > > profile > >> > >> > > > > > > > > responses: > >> > >> > > > > > > > > 200: > >> > >> > > > > > > > > description: OK > >> > >> > > > > > > > > > >> /cluster/flexdown/profile/{profile-name}/{instances}: > >> > >> > > > > > > > > put: > >> > >> > > > > > > > > parameters: > >> > >> > > > > > > > > - name: profile-name > >> > >> > > > > > > > > in: path > >> > >> > > > > > > > > type: string > >> > >> > > > > > > > > description: Instance profile name > >> > >> > > > > > > > > - name: instances > >> > >> > > > > > > > > in: path > >> > >> > > > > > > > > type: integer > >> > >> > > > > > > > > description: Number of instances > >> > >> > > > > > > > > description: Flexdown a multiple instances of > a > >> > >> > specified > >> > >> > > > > > profile > >> > >> > > > > > > > > responses: > >> > >> > > > > > > > > 200: > >> > >> > > > > > > > > description: OK > >> > >> > > > > > > > > /cluster/flexdown/instance/{instance-id}: > >> > >> > > > > > > > > put: > >> > >> > > > > > > > > parameters: > >> > >> > > > > > > > > - name: instance-id > >> > >> > > > > > > > > in: path > >> > >> > > > > > > > > type: string > >> > >> > > > > > > > > description: Instance profile name > >> > >> > > > > > > > > description: Flexup a single instance of a > >> > specified > >> > >> > > > profile > >> > >> > > > > > > > > responses: > >> > >> > > > > > > > > 200: > >> > >> > > > > > > > > description: OK > >> > >> > > > > > > > > definitions: > >> > >> > > > > > > > > FlexUp: > >> > >> > > > > > > > > properties: > >> > >> > > > > > > > > profile: > >> > >> > > > > > > > > type: string > >> > >> > > > > > > > > instances: > >> > >> > > > > > > > > type: integer > >> > >> > > > > > > > > format: int32 > >> > >> > > > > > > > > FlexDown: > >> > >> > > > > > > > > properties: > >> > >> > > > > > > > > instances: > >> > >> > > > > > > > > type: integer > >> > >> > > > > > > > > format: int32 > >> > >> > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella < > >> > >> > > > > > > [email protected]> > >> > >> > > > > > > > > wrote: > >> > >> > > > > > > > > > >> > >> > > > > > > > > > Hi Jim, > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > Did you attach a file? I think the apache > mailing > >> > list > >> > >> > > > > swallowed > >> > >> > > > > > > the > >> > >> > > > > > > > > > attachment. Can you please send us the document > >> > contents > >> > >> in > >> > >> > > > plain > >> > >> > > > > > > text? > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > Thanks, > >> > >> > > > > > > > > > Santosh > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar < > >> > >> > > [email protected]> > >> > >> > > > > > > wrote: > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > This is in regards to > >> > >> > > > > https://github.com/mesos/myriad/issues/89 > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > I created a swagger document (attached) of the > >> > current > >> > >> > flex > >> > >> > > > > APIs > >> > >> > > > > > > and > >> > >> > > > > > > > > what > >> > >> > > > > > > > > > > I propose for the new flex apis. If people like > >> > >> swagger, > >> > >> > I > >> > >> > > > can > >> > >> > > > > > > > document > >> > >> > > > > > > > > > the > >> > >> > > > > > > > > > > rest of the API, including the correct response > >> > codes. > >> > >> > > Either > >> > >> > > > > > way I > >> > >> > > > > > > > > think > >> > >> > > > > > > > > > > we should discuss what the API should be going > >> > >> forward. > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > Paste the file contents into > >> > >> http://editor.swagger.io/ > >> > >> > and > >> > >> > > > > > you'll > >> > >> > > > > > > > get > >> > >> > > > > > > > > > > nice HTML to browse. > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > >> > >> > >> > > > >> > > > >> > > >> > > > > >
