wow, no. I was aware of dpark but I was unaware that they had rolled their own driver. Interesting.
On Wed, Jul 23, 2014 at 11:17 AM, Tom Arnfeld <[email protected]> wrote: > Did anyone know this existed > https://github.com/douban/dpark/tree/master/dpark/pymesos ? Just came > across that while googling... > > > On 23 July 2014 18:28, Erik Erlandson <[email protected]> wrote: > >> >> >> ----- Original Message ----- >> > -1 for git submodules. I am really not keen on those; worked with them >> > while working on Chromium and it was, to be frank, a mess to handle, >> update >> > and maintain. >> > >> >> I've also found submodules disappointing, and been watching on the >> sidelines as the boost community discovers what a pita they are. >> >> A newer alternative is git subtree. Full disclosure: I haven't actually >> worked with subtree, but it looks like a better system than submodules: >> >> http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/ >> >> >> >> > I am rooting for separate repos. Maybe worth a non-binding vote? >> > >> > Niklas >> > >> > >> > On 17 July 2014 11:45, Tim St Clair <[email protected]> wrote: >> > >> > > Inline - >> > > >> > > ------------------------------ >> > > >> > > *From: *"Vladimir Vivien" <[email protected]> >> > > *To: *[email protected] >> > > *Sent: *Tuesday, July 15, 2014 1:34:37 PM >> > > >> > > *Subject: *Re: Mesos language bindings in the wild >> > > >> > > Hi all, >> > > Apologies for being super late to this thread. To answer Niklas >> point at >> > > the start of the thread: Yes, I am thrilled to contribute in anyway I >> can. >> > > The project is moving forward and making progress (slower than I >> want, but >> > > progress regardless). >> > > >> > > Going Native >> > > Implementing a native client for Mesos is an arduous process right now >> > > since there's little doc to guide developers. Once I went through >> C++ code >> > > and a few emails, it became easy (even easier than I thought). If >> the push >> > > is for more native client, at some point we will need basic internals >> to be >> > > documented. >> > > >> > > Mesos-Certified >> > > Maybe a Mesos test suite can be used to certify native clients. >> There are >> > > tons of unit tests in the code that already validate the source code. >> > > Maybe some of those test logic can be pulled out / copied into a >> small >> > > stand-alone mesos test server that clients can communicate with to >> run a >> > > test suite (just an idea). This along with some documentation would >> help >> > > with quality of native clients. >> > > >> > > >> > > +1. >> > > >> > > >> > > In or Out of Core >> > > Having native clients source hosted in core would be great since all >> code >> > > would be in one location. Go code can certainly co-exist a subproject >> in >> > > Mesos. Go's build workflow can be driven by Make. Go's dependency >> > > management can work with repo subdirectories (at least according to >> 'go >> > > help importpath', I haven't tested that myself). But, as Tom pointed >> out, >> > > the thing that raises a flag for me is project velocity. If author >> wants >> > > to move faster or slower than Mesos release cycles, there's no way to >> do so >> > > once the code is part of core. >> > > >> > > Anyway, I have gone on long enough. Looking for ward to feedback. >> > > >> > > >> > > I usually don't tread here, but perhaps a git-submodule works in this >> > > narrow case. >> > > Thoughts? >> > > >> > > >> > > >> > > On Tue, Jul 15, 2014 at 10:07 AM, Tim St Clair <[email protected]> >> > > wrote: >> > > >> > >> Tom - >> > >> >> > >> I understand the desire to create bindings outside the core. The >> point I >> > >> was trying to make earlier around version semantics and testing was >> to >> > >> 'Hedge' the risk. It basically creates a contract between core & >> > >> framework+bindings writers. >> > >> >> > >> No one ever intends to break compatibility, but it happens all the >> time >> > >> and usually in some very subtle ways at first. A great example of >> this is >> > >> a patch I recently submitted to Mesos where the cgroup code was >> writing an >> > >> extra <<endln out. Earlier versions of the kernel had no issue with >> this, >> > >> but recent modifications would cause the cgroup code to fail. Very >> > >> subtle, >> > >> and boom-goes-the-dynamite. >> > >> >> > >> Below was an email I sent a while back, that outlines a possible >> > >> hedge/contract. Please let me know what you think. >> > >> >> > >> -------------------------- >> > >> > >> > >> > Greetings! >> > >> > >> > >> > I've conversed with folks about the idea of having a more >> formalized >> > >> release >> > >> > and branching strategy, such that others who are downstream can >> rely on >> > >> > certain version semantics when planning upgrades, etc. This >> becomes >> > >> doubly >> > >> > important as we start to trend towards a 1.0 release, and folks >> will >> > >> depend >> > >> > heavily on it for their core infrastructure, and APIs (Frameworks, >> and >> > >> EC). >> > >> > >> > >> > Therefore, I wanted to propose a more formalized branching and >> release >> > >> > strategy, and see what others think. I slightly modified this >> pattern >> > >> from >> > >> > the Condor & Kernel projects, which have well established >> processes. >> > >> > >> > >> > ------------------------------ >> > >> > Basic Idea: >> > >> > >> > >> > 1.) Create 2 Main Branches (Stable/Devel-Master based) >> > >> > 2.) Devel releases are cadence/time based and lightly tested. >> > >> > 3.) Stable series only accepts bug fixes. Merge path for all bug >> fixes >> > >> > deemed worthy, are through the stable series up to master. >> > >> > 4.) @ some point devel goes through a *hardning phase* and becomes >> the >> > >> new >> > >> > stable. >> > >> > >> > >> > ------------------------------ >> > >> > Version Semantics: >> > >> > >> > >> > Major.Minor.Revision-PatchBuild >> > >> > >> > >> > Major: >> > >> > - Compatibility breakage (usually protocol or api shift), or >> enough >> > >> minors >> > >> > to justify change. >> > >> >> > >> >> > >> If there is a major version change it should be taken with care and >> > >> notify downstream usually through the >> > >> >> > >> mailing lists. >> > >> >> > >> >> > >> > >> > >> > Minor: >> > >> > - Devel (Odd) - 1.1.x >> > >> > - Stable (Even) - 1.0.x >> > >> > >> > >> > Revision: >> > >> > - Devel - Cadence # Some set of feature enhancements >> > >> > - Stable - Bug and security fixes only (Higher bar of entry) >> > >> > >> > >> > PatchBuild: >> > >> > - Upstream - Whoops our bad, we found a bug or two >> > >> > - Downstream - Back-port build variant. >> > >> > >> > >> > ------------------------------ >> > >> > Series/Branches: >> > >> > >> > >> > Development Series - (Odd Minor #'s): 1.1.x >> > >> > The development series branches/tags are cadence based, and come >> off of >> > >> > master. All new features are added to master. All bug fixes >> should be >> > >> > merged through the stable series into the master. It should be ok >> to >> > >> > introduce destabilizing features from time to time, provided its >> agreed >> > >> upon >> > >> > by a Sheppard. >> > >> > >> > >> > Stable Series - (Even Minor #'s): 1.0.x >> > >> > Stable series should *only contain* bug fixes. This way, >> downstream >> > >> folks >> > >> > have a common understanding that behavior should be maintained. >> Should >> > >> > downstream folks wish to back-port features, they can do that at >> their >> > >> own >> > >> > risk. Every release of the stable series has some measure of >> quality > >> > >> then >> > >> > a +1. E.g. running some clusters for a period of time (X), >> > >> > >> > >> >> > >> >> > >> In this model, stable series should be "stable" for writers against >> the >> > >> API(s). >> > >> >> > >> >> > >> > Transition from Devel-> Stable: >> > >> > After some point, the development series needs to go through a >> hardening >> > >> > phase. This could include static analysis + running on some >> production >> > >> > cluster for a period of time. Folks typically plan the transition >> > >> around a >> > >> > conference series in order to announce the cool new features. >> > >> >> > >> >> > >> + You could test the bindings during this phase ^ but for stable >> series >> > >> they should just work. >> > >> >> > >> >> > >> > ------------------------------ >> > >> >> > >> Cheers, >> > >> Tim >> > >> >> > >> >> > >> ------------------------------ >> > >> >> > >> *From: *"Tom Arnfeld" <[email protected]> >> > >> *To: *[email protected] >> > >> *Sent: *Tuesday, July 15, 2014 2:50:47 AM >> > >> >> > >> *Subject: *Re: Mesos language bindings in the wild >> > >> >> > >> Hey Tim, >> > >> >> > >> I can see your point, and am finding it hard to think of any >> compelling >> > >> arguments against the issue of fragmentation, but I do have a few >> > >> thoughts.p >> > >> >> > >> That said, I would strongly suggest taking ease-of-use and language >> > >> specific code structures into consideration. A huge monolithic build >> > >> system >> > >> might not be a good thing either, if I'm not mistaken that's why >> twitter >> > >> built Pants. >> > >> >> > >> Spark is actually a great example here, it's going to be a huge pain >> to >> > >> publish PySpark to PYPI because of the way they structure the code, >> unless >> > >> you force users to use a bleeding edge version of setuptools to be >> able to >> > >> install the software. In the case of PySpark (and other libraries >> that >> > >> require compiled dependencies, see Hadoofus on github which I >> collaborated >> > >> on this exact issue). It's a nightmare. Projects that work well with >> > >> python setuptools are projects that are just python, from my >> experience. >> > >> >> > >> That said, it's only a nightmare when you *have* precompiled >> dependencies >> > >> that need to be part of the build process. This is no longer the >> case with >> > >> the new mesos bindings, so why make it so hard? >> > >> >> > >> Take Go as another example (this is similar to installing pip >> > >> dependencies from github too) - a user can simply plug in the path >> to a >> > >> repository and away they go. It's easy, and will rapidly speed up >> adoption >> > >> IMO. This isn't something that can easily be done if it's not in >> it's own >> > >> repo, and the Mesos repository is pretty huge now. >> > >> >> > >> My opinion is largely from a users perspective. However, I would ask >> the >> > >> question - how often does the framework API change in such a way >> that it >> > >> breaks compatibility? Will there be a need to orchestrate releases >> among >> > >> 20 >> > >> language bindings to get a release of the core out, how often? Would >> it be >> > >> easier for developers to implement a change and also make that change >> > >> across all languages at the same time, is that even really going to >> > >> happen? >> > >> >> > >> It's also worth considering release cycles, with all bindings being >> built >> > >> into the core, it requires them all the be release together (or it's >> a git >> > >> tag pain). Given that lots of the bindings are going to be (and >> already >> > >> are) community driven, and only a few people are in charge of the >> Mesos >> > >> release cycle (taking at least a few weeks for a release to come >> out) the >> > >> pace for each binding has to be the same, and there's no autonomy. >> > >> >> > >> My personal feeling is that develop user experience isn't thought >> about >> > >> enough is these sorts of situations, and not having a good experience >> > >> either to use or work on the code is a pain and can slow down >> adoption. >> > >> >> > >> Would be interested to hear what you all think, or if you completely >> > >> disagree :-) >> > >> >> > >> Tom. >> > >> >> > >> On Tuesday, 15 July 2014, Tim St Clair <[email protected]> wrote: >> > >> >> > >>> So... your response basically capitulates to the fragmentation >> argument: >> > >>> >> > >>> >> > >>> "Yes we will have binding strewn about of questionable quality that >> may, >> > >>> or may not, work with core." >> > >>> >> > >>> The point that I'm trying to make is, fragmentation *is not* a good >> > >>> thing. >> > >>> >> > >>> -------------------------------------- >> > >>> Case in point - The Hadoop Ecosystem (fragmentation) >> > >>> >> > >>> In order for anyone to make a salient stack of any measure, vendors >> have >> > >>> to knit together components into a stack which can then be consumed >> by >> > >>> the >> > >>> masses. >> > >>> >> > >>> -------------------------------------- >> > >>> Counterpoint - Spark (curating) libraries >> > >>> >> > >>> Spark bundles 1st order interface libraries as part of a curated >> core. >> > >>> You are guaranteed that the core will inter-operate, and PySpark is >> > >>> given >> > >>> 1st class standing. >> > >>> >> > >>> -------------------------------------- >> > >>> >> > >>> This is a bad idea, unless there is a plan to hedge the risk. >> > >>> >> > >>> -Tim >> > >>> >> > >>> >> > >>> ----- Original Message ----- >> > >>> > From: "yifan" <[email protected] <http://JAVASCRIPT-BLOCKED;>> >> > >>> > To: [email protected] <http://JAVASCRIPT-BLOCKED;> >> > >>> > Sent: Monday, July 14, 2014 7:10:34 PM >> > >>> > Subject: Re: Mesos language bindings in the wild >> > >>> > >> > >>> > Hi Tim, >> > >>> > >> > >>> > I found that in zookeeper, they also separate the bindings from >> the >> > >>> core. >> > >>> > >> > >>> > >> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings >> > >>> > >> > >>> > So, IMHO, I think it should be the maintainer's responsibility to >> keep >> > >>> > the binding in healthy state, with clear documentation of which >> version >> > >>> > of the mesos core they supports. >> > >>> > >> > >>> > Yifan >> > >>> > >> > >>> > >> > >>> > >> > >>> > On 07/14/2014 11:30 AM, Tim St Clair wrote: >> > >>> > > So I fear the fragmentation that can occur if we provide native >> > >>> bindings >> > >>> > > outside of the core, unless there is some mechanism for >> testing, & a >> > >>> well >> > >>> > > established versioning scheme. >> > >>> > > >> > >>> > > IMHO, priority inversion on 'versioning' should come before >> bindings >> > >>> to >> > >>> > > ensure we adhere to policy. >> > >>> > > >> > >>> > > Thoughts? >> > >>> > > >> > >>> > > -Tim >> > >>> > > >> > >>> > > >> > >>> > > ----- Original Message ----- >> > >>> > >> From: "Tom Arnfeld" <[email protected] <http://JAVASCRIPT-BLOCKED >> ;>> >> > >>> > >> To: [email protected] <http://JAVASCRIPT-BLOCKED;> >> > >>> > >> Cc: [email protected] <http://JAVASCRIPT-BLOCKED;> >> > >>> > >> Sent: Friday, July 11, 2014 10:22:59 AM >> > >>> > >> Subject: Re: Mesos language bindings in the wild >> > >>> > >> >> > >>> > >> Very exciting. I'd vote +1 for splitting them out. Especially >> if you >> > >>> > >> look at the common way of using Go imports, just stick the >> project >> > >>> on >> > >>> > >> GitHub and import it directly using "github.com/mesos/mesos-go" >> or >> > >>> > >> similar. >> > >>> > >> >> > >>> > >> I guess one argument is that you have more fragmentation of >> the code >> > >>> > >> (e.g every library has it's own copy of the protos) but I'm >> not sure >> > >>> > >> that's a bad thing. >> > >>> > >> >> > >>> > >> Just my two cents. Looking forward to this! >> > >>> > >> >> > >>> > >>> On 11 Jul 2014, at 16:59, Thomas Rampelberg < >> [email protected] >> > >>> <http://JAVASCRIPT-BLOCKED;>> wrote: >> > >>> > >>> >> > >>> > >>> I've started preparing the python bindings to hopefully take >> this >> > >>> > >>> route ( https://reviews.apache.org/r/23224/ would love some >> > >>> reviews! >> > >>> > >>> ). In fact, there is already a native python implementation >> of both >> > >>> > >>> libprocess and the framework apis! ( >> > >>> https://github.com/wickman/pesos/ >> > >>> > >>> , https://github.com/wickman/compactor ). >> > >>> > >>> >> > >>> > >>> What are the benefits of bindings being part of the project >> source >> > >>> > >>> itself instead of having blessed implementations like >> mesos-python >> > >>> > >>> where the source and versioning becomes separate? I've been >> running >> > >>> > >>> into difficulties making automake and python's build tools >> play >> > >>> nicely >> > >>> > >>> together. It seems like there'd be more flexibility in >> general by >> > >>> > >>> splitting them out. >> > >>> > >>> >> > >>> > >>> >> > >>> > >>>> On Thu, Jul 10, 2014 at 3:57 PM, Niklas Nielsen < >> > >>> [email protected] <http://JAVASCRIPT-BLOCKED;>> >> > >>> > >>>> wrote: >> > >>> > >>>> I just wanted to clarify - native, meaning _no_ dependency to >> > >>> libmesos >> > >>> > >>>> and >> > >>> > >>>> native to its language (only Go, only Python and so on) i.e. >> use >> > >>> the >> > >>> > >>>> low-level API. >> > >>> > >>>> >> > >>> > >>>> Sorry for the confusion, >> > >>> > >>>> Niklas >> > >>> > >>>> >> > >>> > >>>> >> > >>> > >>>>> On 10 July 2014 15:55, Dominic Hamon < >> [email protected] >> > >>> <http://JAVASCRIPT-BLOCKED;>> wrote: >> > >>> > >>>>> >> > >>> > >>>>> In my dream world, we wouldn't need any native bindings. I >> can >> > >>> imagine >> > >>> > >>>>> having example frameworks or starter frameworks that use the >> > >>> low-level >> > >>> > >>>>> API >> > >>> > >>>>> (the wire protocol with protocol buffers for message >> passing), >> > >>> but >> > >>> > >>>>> nothing >> > >>> > >>>>> like we have that needs C or JNI, etc. >> > >>> > >>>>> >> > >>> > >>>>> >> > >>> > >>>>> >> > >>> > >>>>> >> > >>> > >>>>> On Thu, Jul 10, 2014 at 3:26 PM, Niklas Nielsen < >> > >>> [email protected] <http://JAVASCRIPT-BLOCKED;>> >> > >>> > >>>>> wrote: >> > >>> > >>>>> >> > >>> > >>>>>> Hi all, >> > >>> > >>>>>> >> > >>> > >>>>>> I wanted to start a discussion around the language >> bindings in >> > >>> the >> > >>> > >>>>>> wild >> > >>> > >>>>>> (Go, Haskell, native Python, Go, Java and so on) and >> possibly >> > >>> get to a >> > >>> > >>>>>> strategy where we start bringing those into Mesos proper. >> As >> > >>> most >> > >>> > >>>>>> things >> > >>> > >>>>>> points towards, it will probably make sense to focus on the >> > >>> native >> > >>> > >>>>>> "bindings" leveraging the low-level API. To name one >> candidate >> > >>> to >> > >>> > >>>>>> start >> > >>> > >>>>>> with, we are especially interested in getting Go native >> support >> > >>> in >> > >>> > >>>>>> Mesos >> > >>> > >>>>>> proper (and in a solid state). So Vladimir, we'd be super >> > >>> thrilled to >> > >>> > >>>>> start >> > >>> > >>>>>> collaborating with you on your current work. >> > >>> > >>>>>> >> > >>> > >>>>>> We are interested to hear what thoughts you all might have >> on >> > >>> this. >> > >>> > >>>>>> >> > >>> > >>>>>> Thanks, >> > >>> > >>>>>> Niklas >> > >>> > >> > >>> > >> > >>> > -- >> > >>> > Gu Yifan >> > >>> > >> > >>> > >> > >>> >> > >>> -- >> > >>> Cheers, >> > >>> Timothy St. Clair >> > >>> Red Hat Inc. >> > >>> >> > >> >> > >> >> > >> >> > >> -- >> > >> Cheers, >> > >> Timothy St. Clair >> > >> Red Hat Inc. >> > >> >> > > >> > > >> > > >> > > -- >> > > Vladimir Vivien >> > > >> > > >> > > >> > > >> > > -- >> > > Cheers, >> > > Timothy St. Clair >> > > Red Hat Inc. >> > > >> > >> > >
