Adoption is good. Competition is good.
On Wed, Jul 23, 2014 at 12:44 PM, Diptanu Choudhury <[email protected]> wrote: > Just saw this - https://github.com/groupon/jesos > > > On Sat, Jul 19, 2014 at 3:22 AM, Vladimir Vivien < > [email protected]> wrote: > >> BenH, >> I created https://issues.apache.org/jira/browse/MESOS-1618 epic to >> capture my thought on that Apache Mesos compatibility test suite for >> framework clients. >> >> >> On Thu, Jul 17, 2014 at 1:29 PM, Benjamin Hindman < >> [email protected]> wrote: >> >>> I love the idea of a Mesos test suite that you could run your framework >>> against. Fantastic! Vladimir, want to create an epic in JIRA for this? >>> >>> Also, and this is likely my fault for the original naming, but I think >>> we should start using the "pure client" nomenclature rather than "native >>> client". >>> >>> >>> On Tue, Jul 15, 2014 at 11:34 AM, Vladimir Vivien < >>> [email protected]> wrote: >>> >>>> Hi all, >>>> Apologies for being super late to this thread. To answer Niklas point >>>> at the start of the thread: Yes, I am thrilled to contribute in anyway I >>>> can. The project is moving forward and making progress (slower than I >>>> want, but progress regardless). >>>> >>>> Going Native >>>> Implementing a native client for Mesos is an arduous process right now >>>> since there's little doc to guide developers. Once I went through C++ code >>>> and a few emails, it became easy (even easier than I thought). If the push >>>> is for more native client, at some point we will need basic internals to be >>>> documented. >>>> >>>> Mesos-Certified >>>> Maybe a Mesos test suite can be used to certify native clients. There >>>> are tons of unit tests in the code that already validate the source code. >>>> Maybe some of those test logic can be pulled out / copied into a small >>>> stand-alone mesos test server that clients can communicate with to run a >>>> test suite (just an idea). This along with some documentation would help >>>> with quality of native clients. >>>> >>>> In or Out of Core >>>> Having native clients source hosted in core would be great since all >>>> code would be in one location. Go code can certainly co-exist a subproject >>>> in Mesos. Go's build workflow can be driven by Make. Go's dependency >>>> management can work with repo subdirectories (at least according to 'go >>>> help importpath', I haven't tested that myself). But, as Tom pointed out, >>>> the thing that raises a flag for me is project velocity. If author wants >>>> to move faster or slower than Mesos release cycles, there's no way to do so >>>> once the code is part of core. >>>> >>>> Anyway, I have gone on long enough. Looking for ward to feedback. >>>> >>>> >>>> On Tue, Jul 15, 2014 at 10:07 AM, Tim St Clair <[email protected]> >>>> wrote: >>>> >>>>> Tom - >>>>> >>>>> I understand the desire to create bindings outside the core. The >>>>> point I was trying to make earlier around version semantics and testing >>>>> was >>>>> to 'Hedge' the risk. It basically creates a contract between core & >>>>> framework+bindings writers. >>>>> >>>>> No one ever intends to break compatibility, but it happens all the >>>>> time and usually in some very subtle ways at first. A great example of >>>>> this is a patch I recently submitted to Mesos where the cgroup code was >>>>> writing an extra <<endln out. Earlier versions of the kernel had no issue >>>>> with this, but recent modifications would cause the cgroup code to fail. >>>>> Very subtle, and boom-goes-the-dynamite. >>>>> >>>>> Below was an email I sent a while back, that outlines a possible >>>>> hedge/contract. Please let me know what you think. >>>>> >>>>> -------------------------- >>>>> > >>>>> > Greetings! >>>>> > >>>>> > I've conversed with folks about the idea of having a more formalized >>>>> release >>>>> > and branching strategy, such that others who are downstream can rely >>>>> on >>>>> > certain version semantics when planning upgrades, etc. This becomes >>>>> doubly >>>>> > important as we start to trend towards a 1.0 release, and folks will >>>>> depend >>>>> > heavily on it for their core infrastructure, and APIs (Frameworks, >>>>> and EC). >>>>> > >>>>> > Therefore, I wanted to propose a more formalized branching and >>>>> release >>>>> > strategy, and see what others think. I slightly modified this >>>>> pattern from >>>>> > the Condor & Kernel projects, which have well established processes. >>>>> > >>>>> > ------------------------------ >>>>> > Basic Idea: >>>>> > >>>>> > 1.) Create 2 Main Branches (Stable/Devel-Master based) >>>>> > 2.) Devel releases are cadence/time based and lightly tested. >>>>> > 3.) Stable series only accepts bug fixes. Merge path for all bug >>>>> fixes >>>>> > deemed worthy, are through the stable series up to master. >>>>> > 4.) @ some point devel goes through a *hardning phase* and becomes >>>>> the new >>>>> > stable. >>>>> > >>>>> > ------------------------------ >>>>> > Version Semantics: >>>>> > >>>>> > Major.Minor.Revision-PatchBuild >>>>> > >>>>> > Major: >>>>> > - Compatibility breakage (usually protocol or api shift), or enough >>>>> minors >>>>> > to justify change. >>>>> >>>>> >>>>> If there is a major version change it should be taken with care and >>>>> notify downstream usually through the >>>>> >>>>> mailing lists. >>>>> >>>>> >>>>> > >>>>> > Minor: >>>>> > - Devel (Odd) - 1.1.x >>>>> > - Stable (Even) - 1.0.x >>>>> > >>>>> > Revision: >>>>> > - Devel - Cadence # Some set of feature enhancements >>>>> > - Stable - Bug and security fixes only (Higher bar of entry) >>>>> > >>>>> > PatchBuild: >>>>> > - Upstream - Whoops our bad, we found a bug or two >>>>> > - Downstream - Back-port build variant. >>>>> > >>>>> > ------------------------------ >>>>> > Series/Branches: >>>>> > >>>>> > Development Series - (Odd Minor #'s): 1.1.x >>>>> > The development series branches/tags are cadence based, and come off >>>>> of >>>>> > master. All new features are added to master. All bug fixes should >>>>> be >>>>> > merged through the stable series into the master. It should be ok to >>>>> > introduce destabilizing features from time to time, provided its >>>>> agreed upon >>>>> > by a Sheppard. >>>>> > >>>>> > Stable Series - (Even Minor #'s): 1.0.x >>>>> > Stable series should *only contain* bug fixes. This way, downstream >>>>> folks >>>>> > have a common understanding that behavior should be maintained. >>>>> Should >>>>> > downstream folks wish to back-port features, they can do that at >>>>> their own >>>>> > risk. Every release of the stable series has some measure of >>>>> quality > then >>>>> > a +1. E.g. running some clusters for a period of time (X), >>>>> > >>>>> >>>>> >>>>> In this model, stable series should be "stable" for writers against >>>>> the API(s). >>>>> >>>>> >>>>> > Transition from Devel-> Stable: >>>>> > After some point, the development series needs to go through a >>>>> hardening >>>>> > phase. This could include static analysis + running on some >>>>> production >>>>> > cluster for a period of time. Folks typically plan the transition >>>>> around a >>>>> > conference series in order to announce the cool new features. >>>>> >>>>> >>>>> + You could test the bindings during this phase ^ but for stable >>>>> series they should just work. >>>>> >>>>> >>>>> > ------------------------------ >>>>> >>>>> Cheers, >>>>> Tim >>>>> >>>>> >>>>> ------------------------------ >>>>> >>>>> *From: *"Tom Arnfeld" <[email protected]> >>>>> *To: *[email protected] >>>>> *Sent: *Tuesday, July 15, 2014 2:50:47 AM >>>>> >>>>> *Subject: *Re: Mesos language bindings in the wild >>>>> >>>>> Hey Tim, >>>>> >>>>> I can see your point, and am finding it hard to think of any >>>>> compelling arguments against the issue of fragmentation, but I do have a >>>>> few thoughts.p >>>>> >>>>> That said, I would strongly suggest taking ease-of-use and language >>>>> specific code structures into consideration. A huge monolithic build >>>>> system >>>>> might not be a good thing either, if I'm not mistaken that's why twitter >>>>> built Pants. >>>>> >>>>> Spark is actually a great example here, it's going to be a huge pain >>>>> to publish PySpark to PYPI because of the way they structure the code, >>>>> unless you force users to use a bleeding edge version of setuptools to be >>>>> able to install the software. In the case of PySpark (and other libraries >>>>> that require compiled dependencies, see Hadoofus on github which I >>>>> collaborated on this exact issue). It's a nightmare. Projects that work >>>>> well with python setuptools are projects that are just python, from my >>>>> experience. >>>>> >>>>> That said, it's only a nightmare when you *have* precompiled >>>>> dependencies that need to be part of the build process. This is no longer >>>>> the case with the new mesos bindings, so why make it so hard? >>>>> >>>>> Take Go as another example (this is similar to installing pip >>>>> dependencies from github too) - a user can simply plug in the path to a >>>>> repository and away they go. It's easy, and will rapidly speed up adoption >>>>> IMO. This isn't something that can easily be done if it's not in it's own >>>>> repo, and the Mesos repository is pretty huge now. >>>>> >>>>> My opinion is largely from a users perspective. However, I would ask >>>>> the question - how often does the framework API change in such a way that >>>>> it breaks compatibility? Will there be a need to orchestrate releases >>>>> among >>>>> 20 language bindings to get a release of the core out, how often? Would it >>>>> be easier for developers to implement a change and also make that change >>>>> across all languages at the same time, is that even really going to >>>>> happen? >>>>> >>>>> It's also worth considering release cycles, with all bindings being >>>>> built into the core, it requires them all the be release together (or it's >>>>> a git tag pain). Given that lots of the bindings are going to be (and >>>>> already are) community driven, and only a few people are in charge of the >>>>> Mesos release cycle (taking at least a few weeks for a release to come >>>>> out) >>>>> the pace for each binding has to be the same, and there's no autonomy. >>>>> >>>>> My personal feeling is that develop user experience isn't thought >>>>> about enough is these sorts of situations, and not having a good >>>>> experience >>>>> either to use or work on the code is a pain and can slow down adoption. >>>>> >>>>> Would be interested to hear what you all think, or if you completely >>>>> disagree :-) >>>>> >>>>> Tom. >>>>> >>>>> On Tuesday, 15 July 2014, Tim St Clair <[email protected]> wrote: >>>>> >>>>>> So... your response basically capitulates to the fragmentation >>>>>> argument: >>>>>> >>>>>> "Yes we will have binding strewn about of questionable quality that >>>>>> may, or may not, work with core." >>>>>> >>>>>> The point that I'm trying to make is, fragmentation *is not* a good >>>>>> thing. >>>>>> >>>>>> -------------------------------------- >>>>>> Case in point - The Hadoop Ecosystem (fragmentation) >>>>>> >>>>>> In order for anyone to make a salient stack of any measure, vendors >>>>>> have to knit together components into a stack which can then be consumed >>>>>> by >>>>>> the masses. >>>>>> >>>>>> -------------------------------------- >>>>>> Counterpoint - Spark (curating) libraries >>>>>> >>>>>> Spark bundles 1st order interface libraries as part of a curated >>>>>> core. You are guaranteed that the core will inter-operate, and PySpark >>>>>> is >>>>>> given 1st class standing. >>>>>> >>>>>> -------------------------------------- >>>>>> >>>>>> This is a bad idea, unless there is a plan to hedge the risk. >>>>>> >>>>>> -Tim >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>> > From: "yifan" <[email protected] <http://JAVASCRIPT-BLOCKED;>> >>>>>> > To: [email protected] <http://JAVASCRIPT-BLOCKED;> >>>>>> > Sent: Monday, July 14, 2014 7:10:34 PM >>>>>> > Subject: Re: Mesos language bindings in the wild >>>>>> > >>>>>> > Hi Tim, >>>>>> > >>>>>> > I found that in zookeeper, they also separate the bindings from the >>>>>> core. >>>>>> > >>>>>> > >>>>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings >>>>>> > >>>>>> > So, IMHO, I think it should be the maintainer's responsibility to >>>>>> keep >>>>>> > the binding in healthy state, with clear documentation of which >>>>>> version >>>>>> > of the mesos core they supports. >>>>>> > >>>>>> > Yifan >>>>>> > >>>>>> > >>>>>> > >>>>>> > On 07/14/2014 11:30 AM, Tim St Clair wrote: >>>>>> > > So I fear the fragmentation that can occur if we provide native >>>>>> bindings >>>>>> > > outside of the core, unless there is some mechanism for testing, >>>>>> & a well >>>>>> > > established versioning scheme. >>>>>> > > >>>>>> > > IMHO, priority inversion on 'versioning' should come before >>>>>> bindings to >>>>>> > > ensure we adhere to policy. >>>>>> > > >>>>>> > > Thoughts? >>>>>> > > >>>>>> > > -Tim >>>>>> > > >>>>>> > > >>>>>> > > ----- Original Message ----- >>>>>> > >> From: "Tom Arnfeld" <[email protected] <http://JAVASCRIPT-BLOCKED;> >>>>>> > >>>>>> > >> To: [email protected] <http://JAVASCRIPT-BLOCKED;> >>>>>> > >> Cc: [email protected] <http://JAVASCRIPT-BLOCKED;> >>>>>> > >> Sent: Friday, July 11, 2014 10:22:59 AM >>>>>> > >> Subject: Re: Mesos language bindings in the wild >>>>>> > >> >>>>>> > >> Very exciting. I'd vote +1 for splitting them out. Especially if >>>>>> you >>>>>> > >> look at the common way of using Go imports, just stick the >>>>>> project on >>>>>> > >> GitHub and import it directly using "github.com/mesos/mesos-go" >>>>>> or >>>>>> > >> similar. >>>>>> > >> >>>>>> > >> I guess one argument is that you have more fragmentation of the >>>>>> code >>>>>> > >> (e.g every library has it's own copy of the protos) but I'm not >>>>>> sure >>>>>> > >> that's a bad thing. >>>>>> > >> >>>>>> > >> Just my two cents. Looking forward to this! >>>>>> > >> >>>>>> > >>> On 11 Jul 2014, at 16:59, Thomas Rampelberg <[email protected] >>>>>> <http://JAVASCRIPT-BLOCKED;>> wrote: >>>>>> > >>> >>>>>> > >>> I've started preparing the python bindings to hopefully take >>>>>> this >>>>>> > >>> route ( https://reviews.apache.org/r/23224/ would love some >>>>>> reviews! >>>>>> > >>> ). In fact, there is already a native python implementation of >>>>>> both >>>>>> > >>> libprocess and the framework apis! ( >>>>>> https://github.com/wickman/pesos/ >>>>>> > >>> , https://github.com/wickman/compactor ). >>>>>> > >>> >>>>>> > >>> What are the benefits of bindings being part of the project >>>>>> source >>>>>> > >>> itself instead of having blessed implementations like >>>>>> mesos-python >>>>>> > >>> where the source and versioning becomes separate? I've been >>>>>> running >>>>>> > >>> into difficulties making automake and python's build tools play >>>>>> nicely >>>>>> > >>> together. It seems like there'd be more flexibility in general >>>>>> by >>>>>> > >>> splitting them out. >>>>>> > >>> >>>>>> > >>> >>>>>> > >>>> On Thu, Jul 10, 2014 at 3:57 PM, Niklas Nielsen < >>>>>> [email protected] <http://JAVASCRIPT-BLOCKED;>> >>>>>> > >>>> wrote: >>>>>> > >>>> I just wanted to clarify - native, meaning _no_ dependency to >>>>>> libmesos >>>>>> > >>>> and >>>>>> > >>>> native to its language (only Go, only Python and so on) i.e. >>>>>> use the >>>>>> > >>>> low-level API. >>>>>> > >>>> >>>>>> > >>>> Sorry for the confusion, >>>>>> > >>>> Niklas >>>>>> > >>>> >>>>>> > >>>> >>>>>> > >>>>> On 10 July 2014 15:55, Dominic Hamon <[email protected] >>>>>> <http://JAVASCRIPT-BLOCKED;>> wrote: >>>>>> > >>>>> >>>>>> > >>>>> In my dream world, we wouldn't need any native bindings. I >>>>>> can imagine >>>>>> > >>>>> having example frameworks or starter frameworks that use the >>>>>> low-level >>>>>> > >>>>> API >>>>>> > >>>>> (the wire protocol with protocol buffers for message >>>>>> passing), but >>>>>> > >>>>> nothing >>>>>> > >>>>> like we have that needs C or JNI, etc. >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> On Thu, Jul 10, 2014 at 3:26 PM, Niklas Nielsen < >>>>>> [email protected] <http://JAVASCRIPT-BLOCKED;>> >>>>>> > >>>>> wrote: >>>>>> > >>>>> >>>>>> > >>>>>> Hi all, >>>>>> > >>>>>> >>>>>> > >>>>>> I wanted to start a discussion around the language bindings >>>>>> in the >>>>>> > >>>>>> wild >>>>>> > >>>>>> (Go, Haskell, native Python, Go, Java and so on) and >>>>>> possibly get to a >>>>>> > >>>>>> strategy where we start bringing those into Mesos proper. As >>>>>> most >>>>>> > >>>>>> things >>>>>> > >>>>>> points towards, it will probably make sense to focus on the >>>>>> native >>>>>> > >>>>>> "bindings" leveraging the low-level API. To name one >>>>>> candidate to >>>>>> > >>>>>> start >>>>>> > >>>>>> with, we are especially interested in getting Go native >>>>>> support in >>>>>> > >>>>>> Mesos >>>>>> > >>>>>> proper (and in a solid state). So Vladimir, we'd be super >>>>>> thrilled to >>>>>> > >>>>> start >>>>>> > >>>>>> collaborating with you on your current work. >>>>>> > >>>>>> >>>>>> > >>>>>> We are interested to hear what thoughts you all might have >>>>>> on this. >>>>>> > >>>>>> >>>>>> > >>>>>> Thanks, >>>>>> > >>>>>> Niklas >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > Gu Yifan >>>>>> > >>>>>> > >>>>>> >>>>>> -- >>>>>> Cheers, >>>>>> Timothy St. Clair >>>>>> Red Hat Inc. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Cheers, >>>>> Timothy St. Clair >>>>> Red Hat Inc. >>>>> >>>> >>>> >>>> >>>> -- >>>> Vladimir Vivien >>>> >>> >>> >> >> >> -- >> Vladimir Vivien >> > > > > -- > Thanks, > Diptanu Choudhury > Web - www.linkedin.com/in/diptanu > Twitter - @diptanu <http://twitter.com/diptanu> > -- Vladimir Vivien
