Just saw this - https://github.com/groupon/jesos
On Sat, Jul 19, 2014 at 3:22 AM, Vladimir Vivien <vladimir.viv...@gmail.com> wrote: > BenH, > I created https://issues.apache.org/jira/browse/MESOS-1618 epic to > capture my thought on that Apache Mesos compatibility test suite for > framework clients. > > > On Thu, Jul 17, 2014 at 1:29 PM, Benjamin Hindman < > benjamin.hind...@gmail.com> wrote: > >> I love the idea of a Mesos test suite that you could run your framework >> against. Fantastic! Vladimir, want to create an epic in JIRA for this? >> >> Also, and this is likely my fault for the original naming, but I think we >> should start using the "pure client" nomenclature rather than "native >> client". >> >> >> On Tue, Jul 15, 2014 at 11:34 AM, Vladimir Vivien < >> vladimir.viv...@gmail.com> wrote: >> >>> Hi all, >>> Apologies for being super late to this thread. To answer Niklas point >>> at the start of the thread: Yes, I am thrilled to contribute in anyway I >>> can. The project is moving forward and making progress (slower than I >>> want, but progress regardless). >>> >>> Going Native >>> Implementing a native client for Mesos is an arduous process right now >>> since there's little doc to guide developers. Once I went through C++ code >>> and a few emails, it became easy (even easier than I thought). If the push >>> is for more native client, at some point we will need basic internals to be >>> documented. >>> >>> Mesos-Certified >>> Maybe a Mesos test suite can be used to certify native clients. There >>> are tons of unit tests in the code that already validate the source code. >>> Maybe some of those test logic can be pulled out / copied into a small >>> stand-alone mesos test server that clients can communicate with to run a >>> test suite (just an idea). This along with some documentation would help >>> with quality of native clients. >>> >>> In or Out of Core >>> Having native clients source hosted in core would be great since all >>> code would be in one location. Go code can certainly co-exist a subproject >>> in Mesos. Go's build workflow can be driven by Make. Go's dependency >>> management can work with repo subdirectories (at least according to 'go >>> help importpath', I haven't tested that myself). But, as Tom pointed out, >>> the thing that raises a flag for me is project velocity. If author wants >>> to move faster or slower than Mesos release cycles, there's no way to do so >>> once the code is part of core. >>> >>> Anyway, I have gone on long enough. Looking for ward to feedback. >>> >>> >>> On Tue, Jul 15, 2014 at 10:07 AM, Tim St Clair <tstcl...@redhat.com> >>> wrote: >>> >>>> Tom - >>>> >>>> I understand the desire to create bindings outside the core. The point >>>> I was trying to make earlier around version semantics and testing was to >>>> 'Hedge' the risk. It basically creates a contract between core & >>>> framework+bindings writers. >>>> >>>> No one ever intends to break compatibility, but it happens all the time >>>> and usually in some very subtle ways at first. A great example of this is >>>> a patch I recently submitted to Mesos where the cgroup code was writing an >>>> extra <<endln out. Earlier versions of the kernel had no issue with this, >>>> but recent modifications would cause the cgroup code to fail. Very subtle, >>>> and boom-goes-the-dynamite. >>>> >>>> Below was an email I sent a while back, that outlines a possible >>>> hedge/contract. Please let me know what you think. >>>> >>>> -------------------------- >>>> > >>>> > Greetings! >>>> > >>>> > I've conversed with folks about the idea of having a more formalized >>>> release >>>> > and branching strategy, such that others who are downstream can rely >>>> on >>>> > certain version semantics when planning upgrades, etc. This becomes >>>> doubly >>>> > important as we start to trend towards a 1.0 release, and folks will >>>> depend >>>> > heavily on it for their core infrastructure, and APIs (Frameworks, >>>> and EC). >>>> > >>>> > Therefore, I wanted to propose a more formalized branching and release >>>> > strategy, and see what others think. I slightly modified this >>>> pattern from >>>> > the Condor & Kernel projects, which have well established processes. >>>> > >>>> > ------------------------------ >>>> > Basic Idea: >>>> > >>>> > 1.) Create 2 Main Branches (Stable/Devel-Master based) >>>> > 2.) Devel releases are cadence/time based and lightly tested. >>>> > 3.) Stable series only accepts bug fixes. Merge path for all bug >>>> fixes >>>> > deemed worthy, are through the stable series up to master. >>>> > 4.) @ some point devel goes through a *hardning phase* and becomes >>>> the new >>>> > stable. >>>> > >>>> > ------------------------------ >>>> > Version Semantics: >>>> > >>>> > Major.Minor.Revision-PatchBuild >>>> > >>>> > Major: >>>> > - Compatibility breakage (usually protocol or api shift), or enough >>>> minors >>>> > to justify change. >>>> >>>> >>>> If there is a major version change it should be taken with care and >>>> notify downstream usually through the >>>> >>>> mailing lists. >>>> >>>> >>>> > >>>> > Minor: >>>> > - Devel (Odd) - 1.1.x >>>> > - Stable (Even) - 1.0.x >>>> > >>>> > Revision: >>>> > - Devel - Cadence # Some set of feature enhancements >>>> > - Stable - Bug and security fixes only (Higher bar of entry) >>>> > >>>> > PatchBuild: >>>> > - Upstream - Whoops our bad, we found a bug or two >>>> > - Downstream - Back-port build variant. >>>> > >>>> > ------------------------------ >>>> > Series/Branches: >>>> > >>>> > Development Series - (Odd Minor #'s): 1.1.x >>>> > The development series branches/tags are cadence based, and come off >>>> of >>>> > master. All new features are added to master. All bug fixes should >>>> be >>>> > merged through the stable series into the master. It should be ok to >>>> > introduce destabilizing features from time to time, provided its >>>> agreed upon >>>> > by a Sheppard. >>>> > >>>> > Stable Series - (Even Minor #'s): 1.0.x >>>> > Stable series should *only contain* bug fixes. This way, downstream >>>> folks >>>> > have a common understanding that behavior should be maintained. >>>> Should >>>> > downstream folks wish to back-port features, they can do that at >>>> their own >>>> > risk. Every release of the stable series has some measure of quality >>>> > then >>>> > a +1. E.g. running some clusters for a period of time (X), >>>> > >>>> >>>> >>>> In this model, stable series should be "stable" for writers against the >>>> API(s). >>>> >>>> >>>> > Transition from Devel-> Stable: >>>> > After some point, the development series needs to go through a >>>> hardening >>>> > phase. This could include static analysis + running on some >>>> production >>>> > cluster for a period of time. Folks typically plan the transition >>>> around a >>>> > conference series in order to announce the cool new features. >>>> >>>> >>>> + You could test the bindings during this phase ^ but for stable series >>>> they should just work. >>>> >>>> >>>> > ------------------------------ >>>> >>>> Cheers, >>>> Tim >>>> >>>> >>>> ------------------------------ >>>> >>>> *From: *"Tom Arnfeld" <t...@duedil.com> >>>> *To: *user@mesos.apache.org >>>> *Sent: *Tuesday, July 15, 2014 2:50:47 AM >>>> >>>> *Subject: *Re: Mesos language bindings in the wild >>>> >>>> Hey Tim, >>>> >>>> I can see your point, and am finding it hard to think of any compelling >>>> arguments against the issue of fragmentation, but I do have a few >>>> thoughts.p >>>> >>>> That said, I would strongly suggest taking ease-of-use and language >>>> specific code structures into consideration. A huge monolithic build system >>>> might not be a good thing either, if I'm not mistaken that's why twitter >>>> built Pants. >>>> >>>> Spark is actually a great example here, it's going to be a huge pain to >>>> publish PySpark to PYPI because of the way they structure the code, unless >>>> you force users to use a bleeding edge version of setuptools to be able to >>>> install the software. In the case of PySpark (and other libraries that >>>> require compiled dependencies, see Hadoofus on github which I collaborated >>>> on this exact issue). It's a nightmare. Projects that work well with >>>> python setuptools are projects that are just python, from my experience. >>>> >>>> That said, it's only a nightmare when you *have* precompiled >>>> dependencies that need to be part of the build process. This is no longer >>>> the case with the new mesos bindings, so why make it so hard? >>>> >>>> Take Go as another example (this is similar to installing pip >>>> dependencies from github too) - a user can simply plug in the path to a >>>> repository and away they go. It's easy, and will rapidly speed up adoption >>>> IMO. This isn't something that can easily be done if it's not in it's own >>>> repo, and the Mesos repository is pretty huge now. >>>> >>>> My opinion is largely from a users perspective. However, I would ask >>>> the question - how often does the framework API change in such a way that >>>> it breaks compatibility? Will there be a need to orchestrate releases among >>>> 20 language bindings to get a release of the core out, how often? Would it >>>> be easier for developers to implement a change and also make that change >>>> across all languages at the same time, is that even really going to happen? >>>> >>>> It's also worth considering release cycles, with all bindings being >>>> built into the core, it requires them all the be release together (or it's >>>> a git tag pain). Given that lots of the bindings are going to be (and >>>> already are) community driven, and only a few people are in charge of the >>>> Mesos release cycle (taking at least a few weeks for a release to come out) >>>> the pace for each binding has to be the same, and there's no autonomy. >>>> >>>> My personal feeling is that develop user experience isn't thought about >>>> enough is these sorts of situations, and not having a good experience >>>> either to use or work on the code is a pain and can slow down adoption. >>>> >>>> Would be interested to hear what you all think, or if you completely >>>> disagree :-) >>>> >>>> Tom. >>>> >>>> On Tuesday, 15 July 2014, Tim St Clair <tstcl...@redhat.com> wrote: >>>> >>>>> So... your response basically capitulates to the fragmentation >>>>> argument: >>>>> >>>>> "Yes we will have binding strewn about of questionable quality that >>>>> may, or may not, work with core." >>>>> >>>>> The point that I'm trying to make is, fragmentation *is not* a good >>>>> thing. >>>>> >>>>> -------------------------------------- >>>>> Case in point - The Hadoop Ecosystem (fragmentation) >>>>> >>>>> In order for anyone to make a salient stack of any measure, vendors >>>>> have to knit together components into a stack which can then be consumed >>>>> by >>>>> the masses. >>>>> >>>>> -------------------------------------- >>>>> Counterpoint - Spark (curating) libraries >>>>> >>>>> Spark bundles 1st order interface libraries as part of a curated core. >>>>> You are guaranteed that the core will inter-operate, and PySpark is given >>>>> 1st class standing. >>>>> >>>>> -------------------------------------- >>>>> >>>>> This is a bad idea, unless there is a plan to hedge the risk. >>>>> >>>>> -Tim >>>>> >>>>> >>>>> ----- Original Message ----- >>>>> > From: "yifan" <myan...@msn.com <http://JAVASCRIPT-BLOCKED;>> >>>>> > To: user@mesos.apache.org <http://JAVASCRIPT-BLOCKED;> >>>>> > Sent: Monday, July 14, 2014 7:10:34 PM >>>>> > Subject: Re: Mesos language bindings in the wild >>>>> > >>>>> > Hi Tim, >>>>> > >>>>> > I found that in zookeeper, they also separate the bindings from the >>>>> core. >>>>> > >>>>> > >>>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings >>>>> > >>>>> > So, IMHO, I think it should be the maintainer's responsibility to >>>>> keep >>>>> > the binding in healthy state, with clear documentation of which >>>>> version >>>>> > of the mesos core they supports. >>>>> > >>>>> > Yifan >>>>> > >>>>> > >>>>> > >>>>> > On 07/14/2014 11:30 AM, Tim St Clair wrote: >>>>> > > So I fear the fragmentation that can occur if we provide native >>>>> bindings >>>>> > > outside of the core, unless there is some mechanism for testing, & >>>>> a well >>>>> > > established versioning scheme. >>>>> > > >>>>> > > IMHO, priority inversion on 'versioning' should come before >>>>> bindings to >>>>> > > ensure we adhere to policy. >>>>> > > >>>>> > > Thoughts? >>>>> > > >>>>> > > -Tim >>>>> > > >>>>> > > >>>>> > > ----- Original Message ----- >>>>> > >> From: "Tom Arnfeld" <t...@duedil.com <http://JAVASCRIPT-BLOCKED;>> >>>>> > >> To: d...@mesos.apache.org <http://JAVASCRIPT-BLOCKED;> >>>>> > >> Cc: user@mesos.apache.org <http://JAVASCRIPT-BLOCKED;> >>>>> > >> Sent: Friday, July 11, 2014 10:22:59 AM >>>>> > >> Subject: Re: Mesos language bindings in the wild >>>>> > >> >>>>> > >> Very exciting. I'd vote +1 for splitting them out. Especially if >>>>> you >>>>> > >> look at the common way of using Go imports, just stick the >>>>> project on >>>>> > >> GitHub and import it directly using "github.com/mesos/mesos-go" >>>>> or >>>>> > >> similar. >>>>> > >> >>>>> > >> I guess one argument is that you have more fragmentation of the >>>>> code >>>>> > >> (e.g every library has it's own copy of the protos) but I'm not >>>>> sure >>>>> > >> that's a bad thing. >>>>> > >> >>>>> > >> Just my two cents. Looking forward to this! >>>>> > >> >>>>> > >>> On 11 Jul 2014, at 16:59, Thomas Rampelberg <tho...@saunter.org >>>>> <http://JAVASCRIPT-BLOCKED;>> wrote: >>>>> > >>> >>>>> > >>> I've started preparing the python bindings to hopefully take this >>>>> > >>> route ( https://reviews.apache.org/r/23224/ would love some >>>>> reviews! >>>>> > >>> ). In fact, there is already a native python implementation of >>>>> both >>>>> > >>> libprocess and the framework apis! ( >>>>> https://github.com/wickman/pesos/ >>>>> > >>> , https://github.com/wickman/compactor ). >>>>> > >>> >>>>> > >>> What are the benefits of bindings being part of the project >>>>> source >>>>> > >>> itself instead of having blessed implementations like >>>>> mesos-python >>>>> > >>> where the source and versioning becomes separate? I've been >>>>> running >>>>> > >>> into difficulties making automake and python's build tools play >>>>> nicely >>>>> > >>> together. It seems like there'd be more flexibility in general by >>>>> > >>> splitting them out. >>>>> > >>> >>>>> > >>> >>>>> > >>>> On Thu, Jul 10, 2014 at 3:57 PM, Niklas Nielsen < >>>>> nik...@mesosphere.io <http://JAVASCRIPT-BLOCKED;>> >>>>> > >>>> wrote: >>>>> > >>>> I just wanted to clarify - native, meaning _no_ dependency to >>>>> libmesos >>>>> > >>>> and >>>>> > >>>> native to its language (only Go, only Python and so on) i.e. >>>>> use the >>>>> > >>>> low-level API. >>>>> > >>>> >>>>> > >>>> Sorry for the confusion, >>>>> > >>>> Niklas >>>>> > >>>> >>>>> > >>>> >>>>> > >>>>> On 10 July 2014 15:55, Dominic Hamon <dha...@twopensource.com >>>>> <http://JAVASCRIPT-BLOCKED;>> wrote: >>>>> > >>>>> >>>>> > >>>>> In my dream world, we wouldn't need any native bindings. I can >>>>> imagine >>>>> > >>>>> having example frameworks or starter frameworks that use the >>>>> low-level >>>>> > >>>>> API >>>>> > >>>>> (the wire protocol with protocol buffers for message passing), >>>>> but >>>>> > >>>>> nothing >>>>> > >>>>> like we have that needs C or JNI, etc. >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> On Thu, Jul 10, 2014 at 3:26 PM, Niklas Nielsen < >>>>> nik...@mesosphere.io <http://JAVASCRIPT-BLOCKED;>> >>>>> > >>>>> wrote: >>>>> > >>>>> >>>>> > >>>>>> Hi all, >>>>> > >>>>>> >>>>> > >>>>>> I wanted to start a discussion around the language bindings >>>>> in the >>>>> > >>>>>> wild >>>>> > >>>>>> (Go, Haskell, native Python, Go, Java and so on) and possibly >>>>> get to a >>>>> > >>>>>> strategy where we start bringing those into Mesos proper. As >>>>> most >>>>> > >>>>>> things >>>>> > >>>>>> points towards, it will probably make sense to focus on the >>>>> native >>>>> > >>>>>> "bindings" leveraging the low-level API. To name one >>>>> candidate to >>>>> > >>>>>> start >>>>> > >>>>>> with, we are especially interested in getting Go native >>>>> support in >>>>> > >>>>>> Mesos >>>>> > >>>>>> proper (and in a solid state). So Vladimir, we'd be super >>>>> thrilled to >>>>> > >>>>> start >>>>> > >>>>>> collaborating with you on your current work. >>>>> > >>>>>> >>>>> > >>>>>> We are interested to hear what thoughts you all might have on >>>>> this. >>>>> > >>>>>> >>>>> > >>>>>> Thanks, >>>>> > >>>>>> Niklas >>>>> > >>>>> > >>>>> > -- >>>>> > Gu Yifan >>>>> > >>>>> > >>>>> >>>>> -- >>>>> Cheers, >>>>> Timothy St. Clair >>>>> Red Hat Inc. >>>>> >>>> >>>> >>>> >>>> -- >>>> Cheers, >>>> Timothy St. Clair >>>> Red Hat Inc. >>>> >>> >>> >>> >>> -- >>> Vladimir Vivien >>> >> >> > > > -- > Vladimir Vivien > -- Thanks, Diptanu Choudhury Web - www.linkedin.com/in/diptanu Twitter - @diptanu <http://twitter.com/diptanu>
