-1 for git submodules. I am really not keen on those; worked with them while working on Chromium and it was, to be frank, a mess to handle, update and maintain.
I am rooting for separate repos. Maybe worth a non-binding vote? Niklas On 17 July 2014 11:45, Tim St Clair <[email protected]> wrote: > Inline - > > ------------------------------ > > *From: *"Vladimir Vivien" <[email protected]> > *To: *[email protected] > *Sent: *Tuesday, July 15, 2014 1:34:37 PM > > *Subject: *Re: Mesos language bindings in the wild > > Hi all, > Apologies for being super late to this thread. To answer Niklas point at > the start of the thread: Yes, I am thrilled to contribute in anyway I can. > The project is moving forward and making progress (slower than I want, but > progress regardless). > > Going Native > Implementing a native client for Mesos is an arduous process right now > since there's little doc to guide developers. Once I went through C++ code > and a few emails, it became easy (even easier than I thought). If the push > is for more native client, at some point we will need basic internals to be > documented. > > Mesos-Certified > Maybe a Mesos test suite can be used to certify native clients. There are > tons of unit tests in the code that already validate the source code. > Maybe some of those test logic can be pulled out / copied into a small > stand-alone mesos test server that clients can communicate with to run a > test suite (just an idea). This along with some documentation would help > with quality of native clients. > > > +1. > > > In or Out of Core > Having native clients source hosted in core would be great since all code > would be in one location. Go code can certainly co-exist a subproject in > Mesos. Go's build workflow can be driven by Make. Go's dependency > management can work with repo subdirectories (at least according to 'go > help importpath', I haven't tested that myself). But, as Tom pointed out, > the thing that raises a flag for me is project velocity. If author wants > to move faster or slower than Mesos release cycles, there's no way to do so > once the code is part of core. > > Anyway, I have gone on long enough. Looking for ward to feedback. > > > I usually don't tread here, but perhaps a git-submodule works in this > narrow case. > Thoughts? > > > > On Tue, Jul 15, 2014 at 10:07 AM, Tim St Clair <[email protected]> > wrote: > >> Tom - >> >> I understand the desire to create bindings outside the core. The point I >> was trying to make earlier around version semantics and testing was to >> 'Hedge' the risk. It basically creates a contract between core & >> framework+bindings writers. >> >> No one ever intends to break compatibility, but it happens all the time >> and usually in some very subtle ways at first. A great example of this is >> a patch I recently submitted to Mesos where the cgroup code was writing an >> extra <<endln out. Earlier versions of the kernel had no issue with this, >> but recent modifications would cause the cgroup code to fail. Very subtle, >> and boom-goes-the-dynamite. >> >> Below was an email I sent a while back, that outlines a possible >> hedge/contract. Please let me know what you think. >> >> -------------------------- >> > >> > Greetings! >> > >> > I've conversed with folks about the idea of having a more formalized >> release >> > and branching strategy, such that others who are downstream can rely on >> > certain version semantics when planning upgrades, etc. This becomes >> doubly >> > important as we start to trend towards a 1.0 release, and folks will >> depend >> > heavily on it for their core infrastructure, and APIs (Frameworks, and >> EC). >> > >> > Therefore, I wanted to propose a more formalized branching and release >> > strategy, and see what others think. I slightly modified this pattern >> from >> > the Condor & Kernel projects, which have well established processes. >> > >> > ------------------------------ >> > Basic Idea: >> > >> > 1.) Create 2 Main Branches (Stable/Devel-Master based) >> > 2.) Devel releases are cadence/time based and lightly tested. >> > 3.) Stable series only accepts bug fixes. Merge path for all bug fixes >> > deemed worthy, are through the stable series up to master. >> > 4.) @ some point devel goes through a *hardning phase* and becomes the >> new >> > stable. >> > >> > ------------------------------ >> > Version Semantics: >> > >> > Major.Minor.Revision-PatchBuild >> > >> > Major: >> > - Compatibility breakage (usually protocol or api shift), or enough >> minors >> > to justify change. >> >> >> If there is a major version change it should be taken with care and >> notify downstream usually through the >> >> mailing lists. >> >> >> > >> > Minor: >> > - Devel (Odd) - 1.1.x >> > - Stable (Even) - 1.0.x >> > >> > Revision: >> > - Devel - Cadence # Some set of feature enhancements >> > - Stable - Bug and security fixes only (Higher bar of entry) >> > >> > PatchBuild: >> > - Upstream - Whoops our bad, we found a bug or two >> > - Downstream - Back-port build variant. >> > >> > ------------------------------ >> > Series/Branches: >> > >> > Development Series - (Odd Minor #'s): 1.1.x >> > The development series branches/tags are cadence based, and come off of >> > master. All new features are added to master. All bug fixes should be >> > merged through the stable series into the master. It should be ok to >> > introduce destabilizing features from time to time, provided its agreed >> upon >> > by a Sheppard. >> > >> > Stable Series - (Even Minor #'s): 1.0.x >> > Stable series should *only contain* bug fixes. This way, downstream >> folks >> > have a common understanding that behavior should be maintained. Should >> > downstream folks wish to back-port features, they can do that at their >> own >> > risk. Every release of the stable series has some measure of quality > >> then >> > a +1. E.g. running some clusters for a period of time (X), >> > >> >> >> In this model, stable series should be "stable" for writers against the >> API(s). >> >> >> > Transition from Devel-> Stable: >> > After some point, the development series needs to go through a hardening >> > phase. This could include static analysis + running on some production >> > cluster for a period of time. Folks typically plan the transition >> around a >> > conference series in order to announce the cool new features. >> >> >> + You could test the bindings during this phase ^ but for stable series >> they should just work. >> >> >> > ------------------------------ >> >> Cheers, >> Tim >> >> >> ------------------------------ >> >> *From: *"Tom Arnfeld" <[email protected]> >> *To: *[email protected] >> *Sent: *Tuesday, July 15, 2014 2:50:47 AM >> >> *Subject: *Re: Mesos language bindings in the wild >> >> Hey Tim, >> >> I can see your point, and am finding it hard to think of any compelling >> arguments against the issue of fragmentation, but I do have a few thoughts.p >> >> That said, I would strongly suggest taking ease-of-use and language >> specific code structures into consideration. A huge monolithic build system >> might not be a good thing either, if I'm not mistaken that's why twitter >> built Pants. >> >> Spark is actually a great example here, it's going to be a huge pain to >> publish PySpark to PYPI because of the way they structure the code, unless >> you force users to use a bleeding edge version of setuptools to be able to >> install the software. In the case of PySpark (and other libraries that >> require compiled dependencies, see Hadoofus on github which I collaborated >> on this exact issue). It's a nightmare. Projects that work well with >> python setuptools are projects that are just python, from my experience. >> >> That said, it's only a nightmare when you *have* precompiled dependencies >> that need to be part of the build process. This is no longer the case with >> the new mesos bindings, so why make it so hard? >> >> Take Go as another example (this is similar to installing pip >> dependencies from github too) - a user can simply plug in the path to a >> repository and away they go. It's easy, and will rapidly speed up adoption >> IMO. This isn't something that can easily be done if it's not in it's own >> repo, and the Mesos repository is pretty huge now. >> >> My opinion is largely from a users perspective. However, I would ask the >> question - how often does the framework API change in such a way that it >> breaks compatibility? Will there be a need to orchestrate releases among 20 >> language bindings to get a release of the core out, how often? Would it be >> easier for developers to implement a change and also make that change >> across all languages at the same time, is that even really going to happen? >> >> It's also worth considering release cycles, with all bindings being built >> into the core, it requires them all the be release together (or it's a git >> tag pain). Given that lots of the bindings are going to be (and already >> are) community driven, and only a few people are in charge of the Mesos >> release cycle (taking at least a few weeks for a release to come out) the >> pace for each binding has to be the same, and there's no autonomy. >> >> My personal feeling is that develop user experience isn't thought about >> enough is these sorts of situations, and not having a good experience >> either to use or work on the code is a pain and can slow down adoption. >> >> Would be interested to hear what you all think, or if you completely >> disagree :-) >> >> Tom. >> >> On Tuesday, 15 July 2014, Tim St Clair <[email protected]> wrote: >> >>> So... your response basically capitulates to the fragmentation argument: >>> >>> >>> "Yes we will have binding strewn about of questionable quality that may, >>> or may not, work with core." >>> >>> The point that I'm trying to make is, fragmentation *is not* a good >>> thing. >>> >>> -------------------------------------- >>> Case in point - The Hadoop Ecosystem (fragmentation) >>> >>> In order for anyone to make a salient stack of any measure, vendors have >>> to knit together components into a stack which can then be consumed by the >>> masses. >>> >>> -------------------------------------- >>> Counterpoint - Spark (curating) libraries >>> >>> Spark bundles 1st order interface libraries as part of a curated core. >>> You are guaranteed that the core will inter-operate, and PySpark is given >>> 1st class standing. >>> >>> -------------------------------------- >>> >>> This is a bad idea, unless there is a plan to hedge the risk. >>> >>> -Tim >>> >>> >>> ----- Original Message ----- >>> > From: "yifan" <[email protected] <http://JAVASCRIPT-BLOCKED;>> >>> > To: [email protected] <http://JAVASCRIPT-BLOCKED;> >>> > Sent: Monday, July 14, 2014 7:10:34 PM >>> > Subject: Re: Mesos language bindings in the wild >>> > >>> > Hi Tim, >>> > >>> > I found that in zookeeper, they also separate the bindings from the >>> core. >>> > >>> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings >>> > >>> > So, IMHO, I think it should be the maintainer's responsibility to keep >>> > the binding in healthy state, with clear documentation of which version >>> > of the mesos core they supports. >>> > >>> > Yifan >>> > >>> > >>> > >>> > On 07/14/2014 11:30 AM, Tim St Clair wrote: >>> > > So I fear the fragmentation that can occur if we provide native >>> bindings >>> > > outside of the core, unless there is some mechanism for testing, & a >>> well >>> > > established versioning scheme. >>> > > >>> > > IMHO, priority inversion on 'versioning' should come before bindings >>> to >>> > > ensure we adhere to policy. >>> > > >>> > > Thoughts? >>> > > >>> > > -Tim >>> > > >>> > > >>> > > ----- Original Message ----- >>> > >> From: "Tom Arnfeld" <[email protected] <http://JAVASCRIPT-BLOCKED;>> >>> > >> To: [email protected] <http://JAVASCRIPT-BLOCKED;> >>> > >> Cc: [email protected] <http://JAVASCRIPT-BLOCKED;> >>> > >> Sent: Friday, July 11, 2014 10:22:59 AM >>> > >> Subject: Re: Mesos language bindings in the wild >>> > >> >>> > >> Very exciting. I'd vote +1 for splitting them out. Especially if you >>> > >> look at the common way of using Go imports, just stick the project >>> on >>> > >> GitHub and import it directly using "github.com/mesos/mesos-go" or >>> > >> similar. >>> > >> >>> > >> I guess one argument is that you have more fragmentation of the code >>> > >> (e.g every library has it's own copy of the protos) but I'm not sure >>> > >> that's a bad thing. >>> > >> >>> > >> Just my two cents. Looking forward to this! >>> > >> >>> > >>> On 11 Jul 2014, at 16:59, Thomas Rampelberg <[email protected] >>> <http://JAVASCRIPT-BLOCKED;>> wrote: >>> > >>> >>> > >>> I've started preparing the python bindings to hopefully take this >>> > >>> route ( https://reviews.apache.org/r/23224/ would love some >>> reviews! >>> > >>> ). In fact, there is already a native python implementation of both >>> > >>> libprocess and the framework apis! ( >>> https://github.com/wickman/pesos/ >>> > >>> , https://github.com/wickman/compactor ). >>> > >>> >>> > >>> What are the benefits of bindings being part of the project source >>> > >>> itself instead of having blessed implementations like mesos-python >>> > >>> where the source and versioning becomes separate? I've been running >>> > >>> into difficulties making automake and python's build tools play >>> nicely >>> > >>> together. It seems like there'd be more flexibility in general by >>> > >>> splitting them out. >>> > >>> >>> > >>> >>> > >>>> On Thu, Jul 10, 2014 at 3:57 PM, Niklas Nielsen < >>> [email protected] <http://JAVASCRIPT-BLOCKED;>> >>> > >>>> wrote: >>> > >>>> I just wanted to clarify - native, meaning _no_ dependency to >>> libmesos >>> > >>>> and >>> > >>>> native to its language (only Go, only Python and so on) i.e. use >>> the >>> > >>>> low-level API. >>> > >>>> >>> > >>>> Sorry for the confusion, >>> > >>>> Niklas >>> > >>>> >>> > >>>> >>> > >>>>> On 10 July 2014 15:55, Dominic Hamon <[email protected] >>> <http://JAVASCRIPT-BLOCKED;>> wrote: >>> > >>>>> >>> > >>>>> In my dream world, we wouldn't need any native bindings. I can >>> imagine >>> > >>>>> having example frameworks or starter frameworks that use the >>> low-level >>> > >>>>> API >>> > >>>>> (the wire protocol with protocol buffers for message passing), >>> but >>> > >>>>> nothing >>> > >>>>> like we have that needs C or JNI, etc. >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> On Thu, Jul 10, 2014 at 3:26 PM, Niklas Nielsen < >>> [email protected] <http://JAVASCRIPT-BLOCKED;>> >>> > >>>>> wrote: >>> > >>>>> >>> > >>>>>> Hi all, >>> > >>>>>> >>> > >>>>>> I wanted to start a discussion around the language bindings in >>> the >>> > >>>>>> wild >>> > >>>>>> (Go, Haskell, native Python, Go, Java and so on) and possibly >>> get to a >>> > >>>>>> strategy where we start bringing those into Mesos proper. As >>> most >>> > >>>>>> things >>> > >>>>>> points towards, it will probably make sense to focus on the >>> native >>> > >>>>>> "bindings" leveraging the low-level API. To name one candidate >>> to >>> > >>>>>> start >>> > >>>>>> with, we are especially interested in getting Go native support >>> in >>> > >>>>>> Mesos >>> > >>>>>> proper (and in a solid state). So Vladimir, we'd be super >>> thrilled to >>> > >>>>> start >>> > >>>>>> collaborating with you on your current work. >>> > >>>>>> >>> > >>>>>> We are interested to hear what thoughts you all might have on >>> this. >>> > >>>>>> >>> > >>>>>> Thanks, >>> > >>>>>> Niklas >>> > >>> > >>> > -- >>> > Gu Yifan >>> > >>> > >>> >>> -- >>> Cheers, >>> Timothy St. Clair >>> Red Hat Inc. >>> >> >> >> >> -- >> Cheers, >> Timothy St. Clair >> Red Hat Inc. >> > > > > -- > Vladimir Vivien > > > > > -- > Cheers, > Timothy St. Clair > Red Hat Inc. >
