I love the idea of a Mesos test suite that you could run your framework against. Fantastic! Vladimir, want to create an epic in JIRA for this?
Also, and this is likely my fault for the original naming, but I think we should start using the "pure client" nomenclature rather than "native client". On Tue, Jul 15, 2014 at 11:34 AM, Vladimir Vivien <vladimir.viv...@gmail.com > wrote: > Hi all, > Apologies for being super late to this thread. To answer Niklas point at > the start of the thread: Yes, I am thrilled to contribute in anyway I can. > The project is moving forward and making progress (slower than I want, but > progress regardless). > > Going Native > Implementing a native client for Mesos is an arduous process right now > since there's little doc to guide developers. Once I went through C++ code > and a few emails, it became easy (even easier than I thought). If the push > is for more native client, at some point we will need basic internals to be > documented. > > Mesos-Certified > Maybe a Mesos test suite can be used to certify native clients. There are > tons of unit tests in the code that already validate the source code. > Maybe some of those test logic can be pulled out / copied into a small > stand-alone mesos test server that clients can communicate with to run a > test suite (just an idea). This along with some documentation would help > with quality of native clients. > > In or Out of Core > Having native clients source hosted in core would be great since all code > would be in one location. Go code can certainly co-exist a subproject in > Mesos. Go's build workflow can be driven by Make. Go's dependency > management can work with repo subdirectories (at least according to 'go > help importpath', I haven't tested that myself). But, as Tom pointed out, > the thing that raises a flag for me is project velocity. If author wants > to move faster or slower than Mesos release cycles, there's no way to do so > once the code is part of core. > > Anyway, I have gone on long enough. Looking for ward to feedback. > > > On Tue, Jul 15, 2014 at 10:07 AM, Tim St Clair <tstcl...@redhat.com> > wrote: > >> Tom - >> >> I understand the desire to create bindings outside the core. The point I >> was trying to make earlier around version semantics and testing was to >> 'Hedge' the risk. It basically creates a contract between core & >> framework+bindings writers. >> >> No one ever intends to break compatibility, but it happens all the time >> and usually in some very subtle ways at first. A great example of this is >> a patch I recently submitted to Mesos where the cgroup code was writing an >> extra <<endln out. Earlier versions of the kernel had no issue with this, >> but recent modifications would cause the cgroup code to fail. Very subtle, >> and boom-goes-the-dynamite. >> >> Below was an email I sent a while back, that outlines a possible >> hedge/contract. Please let me know what you think. >> >> -------------------------- >> > >> > Greetings! >> > >> > I've conversed with folks about the idea of having a more formalized >> release >> > and branching strategy, such that others who are downstream can rely on >> > certain version semantics when planning upgrades, etc. This becomes >> doubly >> > important as we start to trend towards a 1.0 release, and folks will >> depend >> > heavily on it for their core infrastructure, and APIs (Frameworks, and >> EC). >> > >> > Therefore, I wanted to propose a more formalized branching and release >> > strategy, and see what others think. I slightly modified this pattern >> from >> > the Condor & Kernel projects, which have well established processes. >> > >> > ------------------------------ >> > Basic Idea: >> > >> > 1.) Create 2 Main Branches (Stable/Devel-Master based) >> > 2.) Devel releases are cadence/time based and lightly tested. >> > 3.) Stable series only accepts bug fixes. Merge path for all bug fixes >> > deemed worthy, are through the stable series up to master. >> > 4.) @ some point devel goes through a *hardning phase* and becomes the >> new >> > stable. >> > >> > ------------------------------ >> > Version Semantics: >> > >> > Major.Minor.Revision-PatchBuild >> > >> > Major: >> > - Compatibility breakage (usually protocol or api shift), or enough >> minors >> > to justify change. >> >> >> If there is a major version change it should be taken with care and >> notify downstream usually through the >> >> mailing lists. >> >> >> > >> > Minor: >> > - Devel (Odd) - 1.1.x >> > - Stable (Even) - 1.0.x >> > >> > Revision: >> > - Devel - Cadence # Some set of feature enhancements >> > - Stable - Bug and security fixes only (Higher bar of entry) >> > >> > PatchBuild: >> > - Upstream - Whoops our bad, we found a bug or two >> > - Downstream - Back-port build variant. >> > >> > ------------------------------ >> > Series/Branches: >> > >> > Development Series - (Odd Minor #'s): 1.1.x >> > The development series branches/tags are cadence based, and come off of >> > master. All new features are added to master. All bug fixes should be >> > merged through the stable series into the master. It should be ok to >> > introduce destabilizing features from time to time, provided its agreed >> upon >> > by a Sheppard. >> > >> > Stable Series - (Even Minor #'s): 1.0.x >> > Stable series should *only contain* bug fixes. This way, downstream >> folks >> > have a common understanding that behavior should be maintained. Should >> > downstream folks wish to back-port features, they can do that at their >> own >> > risk. Every release of the stable series has some measure of quality > >> then >> > a +1. E.g. running some clusters for a period of time (X), >> > >> >> >> In this model, stable series should be "stable" for writers against the >> API(s). >> >> >> > Transition from Devel-> Stable: >> > After some point, the development series needs to go through a hardening >> > phase. This could include static analysis + running on some production >> > cluster for a period of time. Folks typically plan the transition >> around a >> > conference series in order to announce the cool new features. >> >> >> + You could test the bindings during this phase ^ but for stable series >> they should just work. >> >> >> > ------------------------------ >> >> Cheers, >> Tim >> >> >> ------------------------------ >> >> *From: *"Tom Arnfeld" <t...@duedil.com> >> *To: *user@mesos.apache.org >> *Sent: *Tuesday, July 15, 2014 2:50:47 AM >> >> *Subject: *Re: Mesos language bindings in the wild >> >> Hey Tim, >> >> I can see your point, and am finding it hard to think of any compelling >> arguments against the issue of fragmentation, but I do have a few thoughts.p >> >> That said, I would strongly suggest taking ease-of-use and language >> specific code structures into consideration. A huge monolithic build system >> might not be a good thing either, if I'm not mistaken that's why twitter >> built Pants. >> >> Spark is actually a great example here, it's going to be a huge pain to >> publish PySpark to PYPI because of the way they structure the code, unless >> you force users to use a bleeding edge version of setuptools to be able to >> install the software. In the case of PySpark (and other libraries that >> require compiled dependencies, see Hadoofus on github which I collaborated >> on this exact issue). It's a nightmare. Projects that work well with >> python setuptools are projects that are just python, from my experience. >> >> That said, it's only a nightmare when you *have* precompiled dependencies >> that need to be part of the build process. This is no longer the case with >> the new mesos bindings, so why make it so hard? >> >> Take Go as another example (this is similar to installing pip >> dependencies from github too) - a user can simply plug in the path to a >> repository and away they go. It's easy, and will rapidly speed up adoption >> IMO. This isn't something that can easily be done if it's not in it's own >> repo, and the Mesos repository is pretty huge now. >> >> My opinion is largely from a users perspective. However, I would ask the >> question - how often does the framework API change in such a way that it >> breaks compatibility? Will there be a need to orchestrate releases among 20 >> language bindings to get a release of the core out, how often? Would it be >> easier for developers to implement a change and also make that change >> across all languages at the same time, is that even really going to happen? >> >> It's also worth considering release cycles, with all bindings being built >> into the core, it requires them all the be release together (or it's a git >> tag pain). Given that lots of the bindings are going to be (and already >> are) community driven, and only a few people are in charge of the Mesos >> release cycle (taking at least a few weeks for a release to come out) the >> pace for each binding has to be the same, and there's no autonomy. >> >> My personal feeling is that develop user experience isn't thought about >> enough is these sorts of situations, and not having a good experience >> either to use or work on the code is a pain and can slow down adoption. >> >> Would be interested to hear what you all think, or if you completely >> disagree :-) >> >> Tom. >> >> On Tuesday, 15 July 2014, Tim St Clair <tstcl...@redhat.com> wrote: >> >>> So... your response basically capitulates to the fragmentation argument: >>> >>> >>> "Yes we will have binding strewn about of questionable quality that may, >>> or may not, work with core." >>> >>> The point that I'm trying to make is, fragmentation *is not* a good >>> thing. >>> >>> -------------------------------------- >>> Case in point - The Hadoop Ecosystem (fragmentation) >>> >>> In order for anyone to make a salient stack of any measure, vendors have >>> to knit together components into a stack which can then be consumed by the >>> masses. >>> >>> -------------------------------------- >>> Counterpoint - Spark (curating) libraries >>> >>> Spark bundles 1st order interface libraries as part of a curated core. >>> You are guaranteed that the core will inter-operate, and PySpark is given >>> 1st class standing. >>> >>> -------------------------------------- >>> >>> This is a bad idea, unless there is a plan to hedge the risk. >>> >>> -Tim >>> >>> >>> ----- Original Message ----- >>> > From: "yifan" <myan...@msn.com <http://JAVASCRIPT-BLOCKED;>> >>> > To: user@mesos.apache.org <http://JAVASCRIPT-BLOCKED;> >>> > Sent: Monday, July 14, 2014 7:10:34 PM >>> > Subject: Re: Mesos language bindings in the wild >>> > >>> > Hi Tim, >>> > >>> > I found that in zookeeper, they also separate the bindings from the >>> core. >>> > >>> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings >>> > >>> > So, IMHO, I think it should be the maintainer's responsibility to keep >>> > the binding in healthy state, with clear documentation of which version >>> > of the mesos core they supports. >>> > >>> > Yifan >>> > >>> > >>> > >>> > On 07/14/2014 11:30 AM, Tim St Clair wrote: >>> > > So I fear the fragmentation that can occur if we provide native >>> bindings >>> > > outside of the core, unless there is some mechanism for testing, & a >>> well >>> > > established versioning scheme. >>> > > >>> > > IMHO, priority inversion on 'versioning' should come before bindings >>> to >>> > > ensure we adhere to policy. >>> > > >>> > > Thoughts? >>> > > >>> > > -Tim >>> > > >>> > > >>> > > ----- Original Message ----- >>> > >> From: "Tom Arnfeld" <t...@duedil.com <http://JAVASCRIPT-BLOCKED;>> >>> > >> To: d...@mesos.apache.org <http://JAVASCRIPT-BLOCKED;> >>> > >> Cc: user@mesos.apache.org <http://JAVASCRIPT-BLOCKED;> >>> > >> Sent: Friday, July 11, 2014 10:22:59 AM >>> > >> Subject: Re: Mesos language bindings in the wild >>> > >> >>> > >> Very exciting. I'd vote +1 for splitting them out. Especially if you >>> > >> look at the common way of using Go imports, just stick the project >>> on >>> > >> GitHub and import it directly using "github.com/mesos/mesos-go" or >>> > >> similar. >>> > >> >>> > >> I guess one argument is that you have more fragmentation of the code >>> > >> (e.g every library has it's own copy of the protos) but I'm not sure >>> > >> that's a bad thing. >>> > >> >>> > >> Just my two cents. Looking forward to this! >>> > >> >>> > >>> On 11 Jul 2014, at 16:59, Thomas Rampelberg <tho...@saunter.org >>> <http://JAVASCRIPT-BLOCKED;>> wrote: >>> > >>> >>> > >>> I've started preparing the python bindings to hopefully take this >>> > >>> route ( https://reviews.apache.org/r/23224/ would love some >>> reviews! >>> > >>> ). In fact, there is already a native python implementation of both >>> > >>> libprocess and the framework apis! ( >>> https://github.com/wickman/pesos/ >>> > >>> , https://github.com/wickman/compactor ). >>> > >>> >>> > >>> What are the benefits of bindings being part of the project source >>> > >>> itself instead of having blessed implementations like mesos-python >>> > >>> where the source and versioning becomes separate? I've been running >>> > >>> into difficulties making automake and python's build tools play >>> nicely >>> > >>> together. It seems like there'd be more flexibility in general by >>> > >>> splitting them out. >>> > >>> >>> > >>> >>> > >>>> On Thu, Jul 10, 2014 at 3:57 PM, Niklas Nielsen < >>> nik...@mesosphere.io <http://JAVASCRIPT-BLOCKED;>> >>> > >>>> wrote: >>> > >>>> I just wanted to clarify - native, meaning _no_ dependency to >>> libmesos >>> > >>>> and >>> > >>>> native to its language (only Go, only Python and so on) i.e. use >>> the >>> > >>>> low-level API. >>> > >>>> >>> > >>>> Sorry for the confusion, >>> > >>>> Niklas >>> > >>>> >>> > >>>> >>> > >>>>> On 10 July 2014 15:55, Dominic Hamon <dha...@twopensource.com >>> <http://JAVASCRIPT-BLOCKED;>> wrote: >>> > >>>>> >>> > >>>>> In my dream world, we wouldn't need any native bindings. I can >>> imagine >>> > >>>>> having example frameworks or starter frameworks that use the >>> low-level >>> > >>>>> API >>> > >>>>> (the wire protocol with protocol buffers for message passing), >>> but >>> > >>>>> nothing >>> > >>>>> like we have that needs C or JNI, etc. >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> On Thu, Jul 10, 2014 at 3:26 PM, Niklas Nielsen < >>> nik...@mesosphere.io <http://JAVASCRIPT-BLOCKED;>> >>> > >>>>> wrote: >>> > >>>>> >>> > >>>>>> Hi all, >>> > >>>>>> >>> > >>>>>> I wanted to start a discussion around the language bindings in >>> the >>> > >>>>>> wild >>> > >>>>>> (Go, Haskell, native Python, Go, Java and so on) and possibly >>> get to a >>> > >>>>>> strategy where we start bringing those into Mesos proper. As >>> most >>> > >>>>>> things >>> > >>>>>> points towards, it will probably make sense to focus on the >>> native >>> > >>>>>> "bindings" leveraging the low-level API. To name one candidate >>> to >>> > >>>>>> start >>> > >>>>>> with, we are especially interested in getting Go native support >>> in >>> > >>>>>> Mesos >>> > >>>>>> proper (and in a solid state). So Vladimir, we'd be super >>> thrilled to >>> > >>>>> start >>> > >>>>>> collaborating with you on your current work. >>> > >>>>>> >>> > >>>>>> We are interested to hear what thoughts you all might have on >>> this. >>> > >>>>>> >>> > >>>>>> Thanks, >>> > >>>>>> Niklas >>> > >>> > >>> > -- >>> > Gu Yifan >>> > >>> > >>> >>> -- >>> Cheers, >>> Timothy St. Clair >>> Red Hat Inc. >>> >> >> >> >> -- >> Cheers, >> Timothy St. Clair >> Red Hat Inc. >> > > > > -- > Vladimir Vivien >
