It seems pretty recent, would be interesting to compare notes with pesos/compactor.
On 23 July 2014 19:46, Brian Wickman <wick...@apache.org> wrote: > wow, no. I was aware of dpark but I was unaware that they had rolled > their own driver. Interesting. > > > On Wed, Jul 23, 2014 at 11:17 AM, Tom Arnfeld <t...@duedil.com> wrote: > >> Did anyone know this existed >> https://github.com/douban/dpark/tree/master/dpark/pymesos ? Just came >> across that while googling... >> >> >> On 23 July 2014 18:28, Erik Erlandson <e...@redhat.com> wrote: >> >>> >>> >>> ----- Original Message ----- >>> > -1 for git submodules. I am really not keen on those; worked with them >>> > while working on Chromium and it was, to be frank, a mess to handle, >>> update >>> > and maintain. >>> > >>> >>> I've also found submodules disappointing, and been watching on the >>> sidelines as the boost community discovers what a pita they are. >>> >>> A newer alternative is git subtree. Full disclosure: I haven't actually >>> worked with subtree, but it looks like a better system than submodules: >>> >>> http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/ >>> >>> >>> >>> > I am rooting for separate repos. Maybe worth a non-binding vote? >>> > >>> > Niklas >>> > >>> > >>> > On 17 July 2014 11:45, Tim St Clair <tstcl...@redhat.com> wrote: >>> > >>> > > Inline - >>> > > >>> > > ------------------------------ >>> > > >>> > > *From: *"Vladimir Vivien" <vladimir.viv...@gmail.com> >>> > > *To: *user@mesos.apache.org >>> > > *Sent: *Tuesday, July 15, 2014 1:34:37 PM >>> > > >>> > > *Subject: *Re: Mesos language bindings in the wild >>> > > >>> > > Hi all, >>> > > Apologies for being super late to this thread. To answer Niklas >>> point at >>> > > the start of the thread: Yes, I am thrilled to contribute in anyway >>> I can. >>> > > The project is moving forward and making progress (slower than I >>> want, but >>> > > progress regardless). >>> > > >>> > > Going Native >>> > > Implementing a native client for Mesos is an arduous process right >>> now >>> > > since there's little doc to guide developers. Once I went through >>> C++ code >>> > > and a few emails, it became easy (even easier than I thought). If >>> the push >>> > > is for more native client, at some point we will need basic >>> internals to be >>> > > documented. >>> > > >>> > > Mesos-Certified >>> > > Maybe a Mesos test suite can be used to certify native clients. >>> There are >>> > > tons of unit tests in the code that already validate the source code. >>> > > Maybe some of those test logic can be pulled out / copied into a >>> small >>> > > stand-alone mesos test server that clients can communicate with to >>> run a >>> > > test suite (just an idea). This along with some documentation would >>> help >>> > > with quality of native clients. >>> > > >>> > > >>> > > +1. >>> > > >>> > > >>> > > In or Out of Core >>> > > Having native clients source hosted in core would be great since all >>> code >>> > > would be in one location. Go code can certainly co-exist a >>> subproject in >>> > > Mesos. Go's build workflow can be driven by Make. Go's dependency >>> > > management can work with repo subdirectories (at least according to >>> 'go >>> > > help importpath', I haven't tested that myself). But, as Tom >>> pointed out, >>> > > the thing that raises a flag for me is project velocity. If author >>> wants >>> > > to move faster or slower than Mesos release cycles, there's no way >>> to do so >>> > > once the code is part of core. >>> > > >>> > > Anyway, I have gone on long enough. Looking for ward to feedback. >>> > > >>> > > >>> > > I usually don't tread here, but perhaps a git-submodule works in this >>> > > narrow case. >>> > > Thoughts? >>> > > >>> > > >>> > > >>> > > On Tue, Jul 15, 2014 at 10:07 AM, Tim St Clair <tstcl...@redhat.com> >>> > > wrote: >>> > > >>> > >> Tom - >>> > >> >>> > >> I understand the desire to create bindings outside the core. The >>> point I >>> > >> was trying to make earlier around version semantics and testing was >>> to >>> > >> 'Hedge' the risk. It basically creates a contract between core & >>> > >> framework+bindings writers. >>> > >> >>> > >> No one ever intends to break compatibility, but it happens all the >>> time >>> > >> and usually in some very subtle ways at first. A great example of >>> this is >>> > >> a patch I recently submitted to Mesos where the cgroup code was >>> writing an >>> > >> extra <<endln out. Earlier versions of the kernel had no issue >>> with this, >>> > >> but recent modifications would cause the cgroup code to fail. Very >>> > >> subtle, >>> > >> and boom-goes-the-dynamite. >>> > >> >>> > >> Below was an email I sent a while back, that outlines a possible >>> > >> hedge/contract. Please let me know what you think. >>> > >> >>> > >> -------------------------- >>> > >> > >>> > >> > Greetings! >>> > >> > >>> > >> > I've conversed with folks about the idea of having a more >>> formalized >>> > >> release >>> > >> > and branching strategy, such that others who are downstream can >>> rely on >>> > >> > certain version semantics when planning upgrades, etc. This >>> becomes >>> > >> doubly >>> > >> > important as we start to trend towards a 1.0 release, and folks >>> will >>> > >> depend >>> > >> > heavily on it for their core infrastructure, and APIs >>> (Frameworks, and >>> > >> EC). >>> > >> > >>> > >> > Therefore, I wanted to propose a more formalized branching and >>> release >>> > >> > strategy, and see what others think. I slightly modified this >>> pattern >>> > >> from >>> > >> > the Condor & Kernel projects, which have well established >>> processes. >>> > >> > >>> > >> > ------------------------------ >>> > >> > Basic Idea: >>> > >> > >>> > >> > 1.) Create 2 Main Branches (Stable/Devel-Master based) >>> > >> > 2.) Devel releases are cadence/time based and lightly tested. >>> > >> > 3.) Stable series only accepts bug fixes. Merge path for all bug >>> fixes >>> > >> > deemed worthy, are through the stable series up to master. >>> > >> > 4.) @ some point devel goes through a *hardning phase* and >>> becomes the >>> > >> new >>> > >> > stable. >>> > >> > >>> > >> > ------------------------------ >>> > >> > Version Semantics: >>> > >> > >>> > >> > Major.Minor.Revision-PatchBuild >>> > >> > >>> > >> > Major: >>> > >> > - Compatibility breakage (usually protocol or api shift), or >>> enough >>> > >> minors >>> > >> > to justify change. >>> > >> >>> > >> >>> > >> If there is a major version change it should be taken with care and >>> > >> notify downstream usually through the >>> > >> >>> > >> mailing lists. >>> > >> >>> > >> >>> > >> > >>> > >> > Minor: >>> > >> > - Devel (Odd) - 1.1.x >>> > >> > - Stable (Even) - 1.0.x >>> > >> > >>> > >> > Revision: >>> > >> > - Devel - Cadence # Some set of feature enhancements >>> > >> > - Stable - Bug and security fixes only (Higher bar of entry) >>> > >> > >>> > >> > PatchBuild: >>> > >> > - Upstream - Whoops our bad, we found a bug or two >>> > >> > - Downstream - Back-port build variant. >>> > >> > >>> > >> > ------------------------------ >>> > >> > Series/Branches: >>> > >> > >>> > >> > Development Series - (Odd Minor #'s): 1.1.x >>> > >> > The development series branches/tags are cadence based, and come >>> off of >>> > >> > master. All new features are added to master. All bug fixes >>> should be >>> > >> > merged through the stable series into the master. It should be >>> ok to >>> > >> > introduce destabilizing features from time to time, provided its >>> agreed >>> > >> upon >>> > >> > by a Sheppard. >>> > >> > >>> > >> > Stable Series - (Even Minor #'s): 1.0.x >>> > >> > Stable series should *only contain* bug fixes. This way, >>> downstream >>> > >> folks >>> > >> > have a common understanding that behavior should be maintained. >>> Should >>> > >> > downstream folks wish to back-port features, they can do that at >>> their >>> > >> own >>> > >> > risk. Every release of the stable series has some measure of >>> quality > >>> > >> then >>> > >> > a +1. E.g. running some clusters for a period of time (X), >>> > >> > >>> > >> >>> > >> >>> > >> In this model, stable series should be "stable" for writers against >>> the >>> > >> API(s). >>> > >> >>> > >> >>> > >> > Transition from Devel-> Stable: >>> > >> > After some point, the development series needs to go through a >>> hardening >>> > >> > phase. This could include static analysis + running on some >>> production >>> > >> > cluster for a period of time. Folks typically plan the transition >>> > >> around a >>> > >> > conference series in order to announce the cool new features. >>> > >> >>> > >> >>> > >> + You could test the bindings during this phase ^ but for stable >>> series >>> > >> they should just work. >>> > >> >>> > >> >>> > >> > ------------------------------ >>> > >> >>> > >> Cheers, >>> > >> Tim >>> > >> >>> > >> >>> > >> ------------------------------ >>> > >> >>> > >> *From: *"Tom Arnfeld" <t...@duedil.com> >>> > >> *To: *user@mesos.apache.org >>> > >> *Sent: *Tuesday, July 15, 2014 2:50:47 AM >>> > >> >>> > >> *Subject: *Re: Mesos language bindings in the wild >>> > >> >>> > >> Hey Tim, >>> > >> >>> > >> I can see your point, and am finding it hard to think of any >>> compelling >>> > >> arguments against the issue of fragmentation, but I do have a few >>> > >> thoughts.p >>> > >> >>> > >> That said, I would strongly suggest taking ease-of-use and language >>> > >> specific code structures into consideration. A huge monolithic build >>> > >> system >>> > >> might not be a good thing either, if I'm not mistaken that's why >>> twitter >>> > >> built Pants. >>> > >> >>> > >> Spark is actually a great example here, it's going to be a huge >>> pain to >>> > >> publish PySpark to PYPI because of the way they structure the code, >>> unless >>> > >> you force users to use a bleeding edge version of setuptools to be >>> able to >>> > >> install the software. In the case of PySpark (and other libraries >>> that >>> > >> require compiled dependencies, see Hadoofus on github which I >>> collaborated >>> > >> on this exact issue). It's a nightmare. Projects that work well with >>> > >> python setuptools are projects that are just python, from my >>> experience. >>> > >> >>> > >> That said, it's only a nightmare when you *have* precompiled >>> dependencies >>> > >> that need to be part of the build process. This is no longer the >>> case with >>> > >> the new mesos bindings, so why make it so hard? >>> > >> >>> > >> Take Go as another example (this is similar to installing pip >>> > >> dependencies from github too) - a user can simply plug in the path >>> to a >>> > >> repository and away they go. It's easy, and will rapidly speed up >>> adoption >>> > >> IMO. This isn't something that can easily be done if it's not in >>> it's own >>> > >> repo, and the Mesos repository is pretty huge now. >>> > >> >>> > >> My opinion is largely from a users perspective. However, I would >>> ask the >>> > >> question - how often does the framework API change in such a way >>> that it >>> > >> breaks compatibility? Will there be a need to orchestrate releases >>> among >>> > >> 20 >>> > >> language bindings to get a release of the core out, how often? >>> Would it be >>> > >> easier for developers to implement a change and also make that >>> change >>> > >> across all languages at the same time, is that even really going to >>> > >> happen? >>> > >> >>> > >> It's also worth considering release cycles, with all bindings being >>> built >>> > >> into the core, it requires them all the be release together (or >>> it's a git >>> > >> tag pain). Given that lots of the bindings are going to be (and >>> already >>> > >> are) community driven, and only a few people are in charge of the >>> Mesos >>> > >> release cycle (taking at least a few weeks for a release to come >>> out) the >>> > >> pace for each binding has to be the same, and there's no autonomy. >>> > >> >>> > >> My personal feeling is that develop user experience isn't thought >>> about >>> > >> enough is these sorts of situations, and not having a good >>> experience >>> > >> either to use or work on the code is a pain and can slow down >>> adoption. >>> > >> >>> > >> Would be interested to hear what you all think, or if you completely >>> > >> disagree :-) >>> > >> >>> > >> Tom. >>> > >> >>> > >> On Tuesday, 15 July 2014, Tim St Clair <tstcl...@redhat.com> wrote: >>> > >> >>> > >>> So... your response basically capitulates to the fragmentation >>> argument: >>> > >>> >>> > >>> >>> > >>> "Yes we will have binding strewn about of questionable quality >>> that may, >>> > >>> or may not, work with core." >>> > >>> >>> > >>> The point that I'm trying to make is, fragmentation *is not* a good >>> > >>> thing. >>> > >>> >>> > >>> -------------------------------------- >>> > >>> Case in point - The Hadoop Ecosystem (fragmentation) >>> > >>> >>> > >>> In order for anyone to make a salient stack of any measure, >>> vendors have >>> > >>> to knit together components into a stack which can then be >>> consumed by >>> > >>> the >>> > >>> masses. >>> > >>> >>> > >>> -------------------------------------- >>> > >>> Counterpoint - Spark (curating) libraries >>> > >>> >>> > >>> Spark bundles 1st order interface libraries as part of a curated >>> core. >>> > >>> You are guaranteed that the core will inter-operate, and PySpark >>> is >>> > >>> given >>> > >>> 1st class standing. >>> > >>> >>> > >>> -------------------------------------- >>> > >>> >>> > >>> This is a bad idea, unless there is a plan to hedge the risk. >>> > >>> >>> > >>> -Tim >>> > >>> >>> > >>> >>> > >>> ----- Original Message ----- >>> > >>> > From: "yifan" <myan...@msn.com <http://JAVASCRIPT-BLOCKED;>> >>> > >>> > To: user@mesos.apache.org <http://JAVASCRIPT-BLOCKED;> >>> > >>> > Sent: Monday, July 14, 2014 7:10:34 PM >>> > >>> > Subject: Re: Mesos language bindings in the wild >>> > >>> > >>> > >>> > Hi Tim, >>> > >>> > >>> > >>> > I found that in zookeeper, they also separate the bindings from >>> the >>> > >>> core. >>> > >>> > >>> > >>> > >>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings >>> > >>> > >>> > >>> > So, IMHO, I think it should be the maintainer's responsibility >>> to keep >>> > >>> > the binding in healthy state, with clear documentation of which >>> version >>> > >>> > of the mesos core they supports. >>> > >>> > >>> > >>> > Yifan >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > On 07/14/2014 11:30 AM, Tim St Clair wrote: >>> > >>> > > So I fear the fragmentation that can occur if we provide native >>> > >>> bindings >>> > >>> > > outside of the core, unless there is some mechanism for >>> testing, & a >>> > >>> well >>> > >>> > > established versioning scheme. >>> > >>> > > >>> > >>> > > IMHO, priority inversion on 'versioning' should come before >>> bindings >>> > >>> to >>> > >>> > > ensure we adhere to policy. >>> > >>> > > >>> > >>> > > Thoughts? >>> > >>> > > >>> > >>> > > -Tim >>> > >>> > > >>> > >>> > > >>> > >>> > > ----- Original Message ----- >>> > >>> > >> From: "Tom Arnfeld" <t...@duedil.com < >>> http://JAVASCRIPT-BLOCKED;>> >>> > >>> > >> To: d...@mesos.apache.org <http://JAVASCRIPT-BLOCKED;> >>> > >>> > >> Cc: user@mesos.apache.org <http://JAVASCRIPT-BLOCKED;> >>> > >>> > >> Sent: Friday, July 11, 2014 10:22:59 AM >>> > >>> > >> Subject: Re: Mesos language bindings in the wild >>> > >>> > >> >>> > >>> > >> Very exciting. I'd vote +1 for splitting them out. Especially >>> if you >>> > >>> > >> look at the common way of using Go imports, just stick the >>> project >>> > >>> on >>> > >>> > >> GitHub and import it directly using " >>> github.com/mesos/mesos-go" or >>> > >>> > >> similar. >>> > >>> > >> >>> > >>> > >> I guess one argument is that you have more fragmentation of >>> the code >>> > >>> > >> (e.g every library has it's own copy of the protos) but I'm >>> not sure >>> > >>> > >> that's a bad thing. >>> > >>> > >> >>> > >>> > >> Just my two cents. Looking forward to this! >>> > >>> > >> >>> > >>> > >>> On 11 Jul 2014, at 16:59, Thomas Rampelberg < >>> tho...@saunter.org >>> > >>> <http://JAVASCRIPT-BLOCKED;>> wrote: >>> > >>> > >>> >>> > >>> > >>> I've started preparing the python bindings to hopefully take >>> this >>> > >>> > >>> route ( https://reviews.apache.org/r/23224/ would love some >>> > >>> reviews! >>> > >>> > >>> ). In fact, there is already a native python implementation >>> of both >>> > >>> > >>> libprocess and the framework apis! ( >>> > >>> https://github.com/wickman/pesos/ >>> > >>> > >>> , https://github.com/wickman/compactor ). >>> > >>> > >>> >>> > >>> > >>> What are the benefits of bindings being part of the project >>> source >>> > >>> > >>> itself instead of having blessed implementations like >>> mesos-python >>> > >>> > >>> where the source and versioning becomes separate? I've been >>> running >>> > >>> > >>> into difficulties making automake and python's build tools >>> play >>> > >>> nicely >>> > >>> > >>> together. It seems like there'd be more flexibility in >>> general by >>> > >>> > >>> splitting them out. >>> > >>> > >>> >>> > >>> > >>> >>> > >>> > >>>> On Thu, Jul 10, 2014 at 3:57 PM, Niklas Nielsen < >>> > >>> nik...@mesosphere.io <http://JAVASCRIPT-BLOCKED;>> >>> > >>> > >>>> wrote: >>> > >>> > >>>> I just wanted to clarify - native, meaning _no_ dependency >>> to >>> > >>> libmesos >>> > >>> > >>>> and >>> > >>> > >>>> native to its language (only Go, only Python and so on) >>> i.e. use >>> > >>> the >>> > >>> > >>>> low-level API. >>> > >>> > >>>> >>> > >>> > >>>> Sorry for the confusion, >>> > >>> > >>>> Niklas >>> > >>> > >>>> >>> > >>> > >>>> >>> > >>> > >>>>> On 10 July 2014 15:55, Dominic Hamon < >>> dha...@twopensource.com >>> > >>> <http://JAVASCRIPT-BLOCKED;>> wrote: >>> > >>> > >>>>> >>> > >>> > >>>>> In my dream world, we wouldn't need any native bindings. I >>> can >>> > >>> imagine >>> > >>> > >>>>> having example frameworks or starter frameworks that use >>> the >>> > >>> low-level >>> > >>> > >>>>> API >>> > >>> > >>>>> (the wire protocol with protocol buffers for message >>> passing), >>> > >>> but >>> > >>> > >>>>> nothing >>> > >>> > >>>>> like we have that needs C or JNI, etc. >>> > >>> > >>>>> >>> > >>> > >>>>> >>> > >>> > >>>>> >>> > >>> > >>>>> >>> > >>> > >>>>> On Thu, Jul 10, 2014 at 3:26 PM, Niklas Nielsen < >>> > >>> nik...@mesosphere.io <http://JAVASCRIPT-BLOCKED;>> >>> > >>> > >>>>> wrote: >>> > >>> > >>>>> >>> > >>> > >>>>>> Hi all, >>> > >>> > >>>>>> >>> > >>> > >>>>>> I wanted to start a discussion around the language >>> bindings in >>> > >>> the >>> > >>> > >>>>>> wild >>> > >>> > >>>>>> (Go, Haskell, native Python, Go, Java and so on) and >>> possibly >>> > >>> get to a >>> > >>> > >>>>>> strategy where we start bringing those into Mesos proper. >>> As >>> > >>> most >>> > >>> > >>>>>> things >>> > >>> > >>>>>> points towards, it will probably make sense to focus on >>> the >>> > >>> native >>> > >>> > >>>>>> "bindings" leveraging the low-level API. To name one >>> candidate >>> > >>> to >>> > >>> > >>>>>> start >>> > >>> > >>>>>> with, we are especially interested in getting Go native >>> support >>> > >>> in >>> > >>> > >>>>>> Mesos >>> > >>> > >>>>>> proper (and in a solid state). So Vladimir, we'd be super >>> > >>> thrilled to >>> > >>> > >>>>> start >>> > >>> > >>>>>> collaborating with you on your current work. >>> > >>> > >>>>>> >>> > >>> > >>>>>> We are interested to hear what thoughts you all might >>> have on >>> > >>> this. >>> > >>> > >>>>>> >>> > >>> > >>>>>> Thanks, >>> > >>> > >>>>>> Niklas >>> > >>> > >>> > >>> > >>> > >>> > -- >>> > >>> > Gu Yifan >>> > >>> > >>> > >>> > >>> > >>> >>> > >>> -- >>> > >>> Cheers, >>> > >>> Timothy St. Clair >>> > >>> Red Hat Inc. >>> > >>> >>> > >> >>> > >> >>> > >> >>> > >> -- >>> > >> Cheers, >>> > >> Timothy St. Clair >>> > >> Red Hat Inc. >>> > >> >>> > > >>> > > >>> > > >>> > > -- >>> > > Vladimir Vivien >>> > > >>> > > >>> > > >>> > > >>> > > -- >>> > > Cheers, >>> > > Timothy St. Clair >>> > > Red Hat Inc. >>> > > >>> > >>> >> >> >
