Soon, status update acknowledgements must be sent back to the master, rather than to the slave directly.
0.19.x supports sending them back to the master, so if you're not aiming to support any older versions of mesos this line could be updated: https://github.com/tarnfeld/pesos/blob/master/src/pesos/scheduler.py#L166 On Tue, Jul 29, 2014 at 6:48 AM, Tom Arnfeld <[email protected]> wrote: > Figured i'd ping round an update on this thread if anyone is interested. > I've been working on @wickman's pesos/compactor libraries as I mentioned > before, and I've now got pesos to a state that i'm able to consistently > launch frameworks and executors using the pure python bindings. > > I think i've implemented all of the framework scheduler/executor APIs > (including framework/executor messages) – though the implementation for ZK > groups is missing (i'll be using the twitter.common library for that) and > if the master fails over, the framework will currently not behave correctly. > > If anyone has time to give my branches a try ( > github.com/tarnfeld/compactor and github.com/tarnfeld/pesos) that'd be > awesome, there is definitely a fair way to go still to get feature parity > with the existing native API. I've updated my little test python framework > https://github.com/tarnfeld/mesos-python-framework to use pesos, as a > working example. Sorry the docs are a little out of date on that one. > > Tom. > > > On 23 July 2014 19:48, Tom Arnfeld <[email protected]> wrote: > >> It seems pretty recent, would be interesting to compare notes with >> pesos/compactor. >> >> >> On 23 July 2014 19:46, Brian Wickman <[email protected]> wrote: >> >>> wow, no. I was aware of dpark but I was unaware that they had rolled >>> their own driver. Interesting. >>> >>> >>> On Wed, Jul 23, 2014 at 11:17 AM, Tom Arnfeld <[email protected]> wrote: >>> >>>> Did anyone know this existed >>>> https://github.com/douban/dpark/tree/master/dpark/pymesos ? Just came >>>> across that while googling... >>>> >>>> >>>> On 23 July 2014 18:28, Erik Erlandson <[email protected]> wrote: >>>> >>>>> >>>>> >>>>> ----- Original Message ----- >>>>> > -1 for git submodules. I am really not keen on those; worked with >>>>> them >>>>> > while working on Chromium and it was, to be frank, a mess to handle, >>>>> update >>>>> > and maintain. >>>>> > >>>>> >>>>> I've also found submodules disappointing, and been watching on the >>>>> sidelines as the boost community discovers what a pita they are. >>>>> >>>>> A newer alternative is git subtree. Full disclosure: I haven't >>>>> actually worked with subtree, but it looks like a better system than >>>>> submodules: >>>>> >>>>> http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/ >>>>> >>>>> >>>>> >>>>> > I am rooting for separate repos. Maybe worth a non-binding vote? >>>>> > >>>>> > Niklas >>>>> > >>>>> > >>>>> > On 17 July 2014 11:45, Tim St Clair <[email protected]> wrote: >>>>> > >>>>> > > Inline - >>>>> > > >>>>> > > ------------------------------ >>>>> > > >>>>> > > *From: *"Vladimir Vivien" <[email protected]> >>>>> > > *To: *[email protected] >>>>> > > *Sent: *Tuesday, July 15, 2014 1:34:37 PM >>>>> > > >>>>> > > *Subject: *Re: Mesos language bindings in the wild >>>>> > > >>>>> > > Hi all, >>>>> > > Apologies for being super late to this thread. To answer Niklas >>>>> point at >>>>> > > the start of the thread: Yes, I am thrilled to contribute in >>>>> anyway I can. >>>>> > > The project is moving forward and making progress (slower than I >>>>> want, but >>>>> > > progress regardless). >>>>> > > >>>>> > > Going Native >>>>> > > Implementing a native client for Mesos is an arduous process right >>>>> now >>>>> > > since there's little doc to guide developers. Once I went through >>>>> C++ code >>>>> > > and a few emails, it became easy (even easier than I thought). If >>>>> the push >>>>> > > is for more native client, at some point we will need basic >>>>> internals to be >>>>> > > documented. >>>>> > > >>>>> > > Mesos-Certified >>>>> > > Maybe a Mesos test suite can be used to certify native clients. >>>>> There are >>>>> > > tons of unit tests in the code that already validate the source >>>>> code. >>>>> > > Maybe some of those test logic can be pulled out / copied into a >>>>> small >>>>> > > stand-alone mesos test server that clients can communicate with to >>>>> run a >>>>> > > test suite (just an idea). This along with some documentation >>>>> would help >>>>> > > with quality of native clients. >>>>> > > >>>>> > > >>>>> > > +1. >>>>> > > >>>>> > > >>>>> > > In or Out of Core >>>>> > > Having native clients source hosted in core would be great since >>>>> all code >>>>> > > would be in one location. Go code can certainly co-exist a >>>>> subproject in >>>>> > > Mesos. Go's build workflow can be driven by Make. Go's dependency >>>>> > > management can work with repo subdirectories (at least according >>>>> to 'go >>>>> > > help importpath', I haven't tested that myself). But, as Tom >>>>> pointed out, >>>>> > > the thing that raises a flag for me is project velocity. If >>>>> author wants >>>>> > > to move faster or slower than Mesos release cycles, there's no way >>>>> to do so >>>>> > > once the code is part of core. >>>>> > > >>>>> > > Anyway, I have gone on long enough. Looking for ward to feedback. >>>>> > > >>>>> > > >>>>> > > I usually don't tread here, but perhaps a git-submodule works in >>>>> this >>>>> > > narrow case. >>>>> > > Thoughts? >>>>> > > >>>>> > > >>>>> > > >>>>> > > On Tue, Jul 15, 2014 at 10:07 AM, Tim St Clair < >>>>> [email protected]> >>>>> > > wrote: >>>>> > > >>>>> > >> Tom - >>>>> > >> >>>>> > >> I understand the desire to create bindings outside the core. The >>>>> point I >>>>> > >> was trying to make earlier around version semantics and testing >>>>> was to >>>>> > >> 'Hedge' the risk. It basically creates a contract between core & >>>>> > >> framework+bindings writers. >>>>> > >> >>>>> > >> No one ever intends to break compatibility, but it happens all >>>>> the time >>>>> > >> and usually in some very subtle ways at first. A great example >>>>> of this is >>>>> > >> a patch I recently submitted to Mesos where the cgroup code was >>>>> writing an >>>>> > >> extra <<endln out. Earlier versions of the kernel had no issue >>>>> with this, >>>>> > >> but recent modifications would cause the cgroup code to fail. >>>>> Very >>>>> > >> subtle, >>>>> > >> and boom-goes-the-dynamite. >>>>> > >> >>>>> > >> Below was an email I sent a while back, that outlines a possible >>>>> > >> hedge/contract. Please let me know what you think. >>>>> > >> >>>>> > >> -------------------------- >>>>> > >> > >>>>> > >> > Greetings! >>>>> > >> > >>>>> > >> > I've conversed with folks about the idea of having a more >>>>> formalized >>>>> > >> release >>>>> > >> > and branching strategy, such that others who are downstream can >>>>> rely on >>>>> > >> > certain version semantics when planning upgrades, etc. This >>>>> becomes >>>>> > >> doubly >>>>> > >> > important as we start to trend towards a 1.0 release, and folks >>>>> will >>>>> > >> depend >>>>> > >> > heavily on it for their core infrastructure, and APIs >>>>> (Frameworks, and >>>>> > >> EC). >>>>> > >> > >>>>> > >> > Therefore, I wanted to propose a more formalized branching and >>>>> release >>>>> > >> > strategy, and see what others think. I slightly modified this >>>>> pattern >>>>> > >> from >>>>> > >> > the Condor & Kernel projects, which have well established >>>>> processes. >>>>> > >> > >>>>> > >> > ------------------------------ >>>>> > >> > Basic Idea: >>>>> > >> > >>>>> > >> > 1.) Create 2 Main Branches (Stable/Devel-Master based) >>>>> > >> > 2.) Devel releases are cadence/time based and lightly tested. >>>>> > >> > 3.) Stable series only accepts bug fixes. Merge path for all >>>>> bug fixes >>>>> > >> > deemed worthy, are through the stable series up to master. >>>>> > >> > 4.) @ some point devel goes through a *hardning phase* and >>>>> becomes the >>>>> > >> new >>>>> > >> > stable. >>>>> > >> > >>>>> > >> > ------------------------------ >>>>> > >> > Version Semantics: >>>>> > >> > >>>>> > >> > Major.Minor.Revision-PatchBuild >>>>> > >> > >>>>> > >> > Major: >>>>> > >> > - Compatibility breakage (usually protocol or api shift), or >>>>> enough >>>>> > >> minors >>>>> > >> > to justify change. >>>>> > >> >>>>> > >> >>>>> > >> If there is a major version change it should be taken with care >>>>> and >>>>> > >> notify downstream usually through the >>>>> > >> >>>>> > >> mailing lists. >>>>> > >> >>>>> > >> >>>>> > >> > >>>>> > >> > Minor: >>>>> > >> > - Devel (Odd) - 1.1.x >>>>> > >> > - Stable (Even) - 1.0.x >>>>> > >> > >>>>> > >> > Revision: >>>>> > >> > - Devel - Cadence # Some set of feature enhancements >>>>> > >> > - Stable - Bug and security fixes only (Higher bar of entry) >>>>> > >> > >>>>> > >> > PatchBuild: >>>>> > >> > - Upstream - Whoops our bad, we found a bug or two >>>>> > >> > - Downstream - Back-port build variant. >>>>> > >> > >>>>> > >> > ------------------------------ >>>>> > >> > Series/Branches: >>>>> > >> > >>>>> > >> > Development Series - (Odd Minor #'s): 1.1.x >>>>> > >> > The development series branches/tags are cadence based, and >>>>> come off of >>>>> > >> > master. All new features are added to master. All bug fixes >>>>> should be >>>>> > >> > merged through the stable series into the master. It should be >>>>> ok to >>>>> > >> > introduce destabilizing features from time to time, provided >>>>> its agreed >>>>> > >> upon >>>>> > >> > by a Sheppard. >>>>> > >> > >>>>> > >> > Stable Series - (Even Minor #'s): 1.0.x >>>>> > >> > Stable series should *only contain* bug fixes. This way, >>>>> downstream >>>>> > >> folks >>>>> > >> > have a common understanding that behavior should be maintained. >>>>> Should >>>>> > >> > downstream folks wish to back-port features, they can do that >>>>> at their >>>>> > >> own >>>>> > >> > risk. Every release of the stable series has some measure of >>>>> quality > >>>>> > >> then >>>>> > >> > a +1. E.g. running some clusters for a period of time (X), >>>>> > >> > >>>>> > >> >>>>> > >> >>>>> > >> In this model, stable series should be "stable" for writers >>>>> against the >>>>> > >> API(s). >>>>> > >> >>>>> > >> >>>>> > >> > Transition from Devel-> Stable: >>>>> > >> > After some point, the development series needs to go through a >>>>> hardening >>>>> > >> > phase. This could include static analysis + running on some >>>>> production >>>>> > >> > cluster for a period of time. Folks typically plan the >>>>> transition >>>>> > >> around a >>>>> > >> > conference series in order to announce the cool new features. >>>>> > >> >>>>> > >> >>>>> > >> + You could test the bindings during this phase ^ but for stable >>>>> series >>>>> > >> they should just work. >>>>> > >> >>>>> > >> >>>>> > >> > ------------------------------ >>>>> > >> >>>>> > >> Cheers, >>>>> > >> Tim >>>>> > >> >>>>> > >> >>>>> > >> ------------------------------ >>>>> > >> >>>>> > >> *From: *"Tom Arnfeld" <[email protected]> >>>>> > >> *To: *[email protected] >>>>> > >> *Sent: *Tuesday, July 15, 2014 2:50:47 AM >>>>> > >> >>>>> > >> *Subject: *Re: Mesos language bindings in the wild >>>>> > >> >>>>> > >> Hey Tim, >>>>> > >> >>>>> > >> I can see your point, and am finding it hard to think of any >>>>> compelling >>>>> > >> arguments against the issue of fragmentation, but I do have a few >>>>> > >> thoughts.p >>>>> > >> >>>>> > >> That said, I would strongly suggest taking ease-of-use and >>>>> language >>>>> > >> specific code structures into consideration. A huge monolithic >>>>> build >>>>> > >> system >>>>> > >> might not be a good thing either, if I'm not mistaken that's why >>>>> twitter >>>>> > >> built Pants. >>>>> > >> >>>>> > >> Spark is actually a great example here, it's going to be a huge >>>>> pain to >>>>> > >> publish PySpark to PYPI because of the way they structure the >>>>> code, unless >>>>> > >> you force users to use a bleeding edge version of setuptools to >>>>> be able to >>>>> > >> install the software. In the case of PySpark (and other libraries >>>>> that >>>>> > >> require compiled dependencies, see Hadoofus on github which I >>>>> collaborated >>>>> > >> on this exact issue). It's a nightmare. Projects that work well >>>>> with >>>>> > >> python setuptools are projects that are just python, from my >>>>> experience. >>>>> > >> >>>>> > >> That said, it's only a nightmare when you *have* precompiled >>>>> dependencies >>>>> > >> that need to be part of the build process. This is no longer the >>>>> case with >>>>> > >> the new mesos bindings, so why make it so hard? >>>>> > >> >>>>> > >> Take Go as another example (this is similar to installing pip >>>>> > >> dependencies from github too) - a user can simply plug in the >>>>> path to a >>>>> > >> repository and away they go. It's easy, and will rapidly speed up >>>>> adoption >>>>> > >> IMO. This isn't something that can easily be done if it's not in >>>>> it's own >>>>> > >> repo, and the Mesos repository is pretty huge now. >>>>> > >> >>>>> > >> My opinion is largely from a users perspective. However, I would >>>>> ask the >>>>> > >> question - how often does the framework API change in such a way >>>>> that it >>>>> > >> breaks compatibility? Will there be a need to orchestrate >>>>> releases among >>>>> > >> 20 >>>>> > >> language bindings to get a release of the core out, how often? >>>>> Would it be >>>>> > >> easier for developers to implement a change and also make that >>>>> change >>>>> > >> across all languages at the same time, is that even really going >>>>> to >>>>> > >> happen? >>>>> > >> >>>>> > >> It's also worth considering release cycles, with all bindings >>>>> being built >>>>> > >> into the core, it requires them all the be release together (or >>>>> it's a git >>>>> > >> tag pain). Given that lots of the bindings are going to be (and >>>>> already >>>>> > >> are) community driven, and only a few people are in charge of the >>>>> Mesos >>>>> > >> release cycle (taking at least a few weeks for a release to come >>>>> out) the >>>>> > >> pace for each binding has to be the same, and there's no autonomy. >>>>> > >> >>>>> > >> My personal feeling is that develop user experience isn't thought >>>>> about >>>>> > >> enough is these sorts of situations, and not having a good >>>>> experience >>>>> > >> either to use or work on the code is a pain and can slow down >>>>> adoption. >>>>> > >> >>>>> > >> Would be interested to hear what you all think, or if you >>>>> completely >>>>> > >> disagree :-) >>>>> > >> >>>>> > >> Tom. >>>>> > >> >>>>> > >> On Tuesday, 15 July 2014, Tim St Clair <[email protected]> >>>>> wrote: >>>>> > >> >>>>> > >>> So... your response basically capitulates to the fragmentation >>>>> argument: >>>>> > >>> >>>>> > >>> >>>>> > >>> "Yes we will have binding strewn about of questionable quality >>>>> that may, >>>>> > >>> or may not, work with core." >>>>> > >>> >>>>> > >>> The point that I'm trying to make is, fragmentation *is not* a >>>>> good >>>>> > >>> thing. >>>>> > >>> >>>>> > >>> -------------------------------------- >>>>> > >>> Case in point - The Hadoop Ecosystem (fragmentation) >>>>> > >>> >>>>> > >>> In order for anyone to make a salient stack of any measure, >>>>> vendors have >>>>> > >>> to knit together components into a stack which can then be >>>>> consumed by >>>>> > >>> the >>>>> > >>> masses. >>>>> > >>> >>>>> > >>> -------------------------------------- >>>>> > >>> Counterpoint - Spark (curating) libraries >>>>> > >>> >>>>> > >>> Spark bundles 1st order interface libraries as part of a curated >>>>> core. >>>>> > >>> You are guaranteed that the core will inter-operate, and >>>>> PySpark is >>>>> > >>> given >>>>> > >>> 1st class standing. >>>>> > >>> >>>>> > >>> -------------------------------------- >>>>> > >>> >>>>> > >>> This is a bad idea, unless there is a plan to hedge the risk. >>>>> > >>> >>>>> > >>> -Tim >>>>> > >>> >>>>> > >>> >>>>> > >>> ----- Original Message ----- >>>>> > >>> > From: "yifan" <[email protected] <http://JAVASCRIPT-BLOCKED;>> >>>>> > >>> > To: [email protected] <http://JAVASCRIPT-BLOCKED;> >>>>> > >>> > Sent: Monday, July 14, 2014 7:10:34 PM >>>>> > >>> > Subject: Re: Mesos language bindings in the wild >>>>> > >>> > >>>>> > >>> > Hi Tim, >>>>> > >>> > >>>>> > >>> > I found that in zookeeper, they also separate the bindings >>>>> from the >>>>> > >>> core. >>>>> > >>> > >>>>> > >>> > >>>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings >>>>> > >>> > >>>>> > >>> > So, IMHO, I think it should be the maintainer's responsibility >>>>> to keep >>>>> > >>> > the binding in healthy state, with clear documentation of >>>>> which version >>>>> > >>> > of the mesos core they supports. >>>>> > >>> > >>>>> > >>> > Yifan >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > On 07/14/2014 11:30 AM, Tim St Clair wrote: >>>>> > >>> > > So I fear the fragmentation that can occur if we provide >>>>> native >>>>> > >>> bindings >>>>> > >>> > > outside of the core, unless there is some mechanism for >>>>> testing, & a >>>>> > >>> well >>>>> > >>> > > established versioning scheme. >>>>> > >>> > > >>>>> > >>> > > IMHO, priority inversion on 'versioning' should come before >>>>> bindings >>>>> > >>> to >>>>> > >>> > > ensure we adhere to policy. >>>>> > >>> > > >>>>> > >>> > > Thoughts? >>>>> > >>> > > >>>>> > >>> > > -Tim >>>>> > >>> > > >>>>> > >>> > > >>>>> > >>> > > ----- Original Message ----- >>>>> > >>> > >> From: "Tom Arnfeld" <[email protected] < >>>>> http://JAVASCRIPT-BLOCKED;>> >>>>> > >>> > >> To: [email protected] <http://JAVASCRIPT-BLOCKED;> >>>>> > >>> > >> Cc: [email protected] <http://JAVASCRIPT-BLOCKED;> >>>>> > >>> > >> Sent: Friday, July 11, 2014 10:22:59 AM >>>>> > >>> > >> Subject: Re: Mesos language bindings in the wild >>>>> > >>> > >> >>>>> > >>> > >> Very exciting. I'd vote +1 for splitting them out. >>>>> Especially if you >>>>> > >>> > >> look at the common way of using Go imports, just stick the >>>>> project >>>>> > >>> on >>>>> > >>> > >> GitHub and import it directly using " >>>>> github.com/mesos/mesos-go" or >>>>> > >>> > >> similar. >>>>> > >>> > >> >>>>> > >>> > >> I guess one argument is that you have more fragmentation of >>>>> the code >>>>> > >>> > >> (e.g every library has it's own copy of the protos) but I'm >>>>> not sure >>>>> > >>> > >> that's a bad thing. >>>>> > >>> > >> >>>>> > >>> > >> Just my two cents. Looking forward to this! >>>>> > >>> > >> >>>>> > >>> > >>> On 11 Jul 2014, at 16:59, Thomas Rampelberg < >>>>> [email protected] >>>>> > >>> <http://JAVASCRIPT-BLOCKED;>> wrote: >>>>> > >>> > >>> >>>>> > >>> > >>> I've started preparing the python bindings to hopefully >>>>> take this >>>>> > >>> > >>> route ( https://reviews.apache.org/r/23224/ would love >>>>> some >>>>> > >>> reviews! >>>>> > >>> > >>> ). In fact, there is already a native python >>>>> implementation of both >>>>> > >>> > >>> libprocess and the framework apis! ( >>>>> > >>> https://github.com/wickman/pesos/ >>>>> > >>> > >>> , https://github.com/wickman/compactor ). >>>>> > >>> > >>> >>>>> > >>> > >>> What are the benefits of bindings being part of the >>>>> project source >>>>> > >>> > >>> itself instead of having blessed implementations like >>>>> mesos-python >>>>> > >>> > >>> where the source and versioning becomes separate? I've >>>>> been running >>>>> > >>> > >>> into difficulties making automake and python's build tools >>>>> play >>>>> > >>> nicely >>>>> > >>> > >>> together. It seems like there'd be more flexibility in >>>>> general by >>>>> > >>> > >>> splitting them out. >>>>> > >>> > >>> >>>>> > >>> > >>> >>>>> > >>> > >>>> On Thu, Jul 10, 2014 at 3:57 PM, Niklas Nielsen < >>>>> > >>> [email protected] <http://JAVASCRIPT-BLOCKED;>> >>>>> > >>> > >>>> wrote: >>>>> > >>> > >>>> I just wanted to clarify - native, meaning _no_ >>>>> dependency to >>>>> > >>> libmesos >>>>> > >>> > >>>> and >>>>> > >>> > >>>> native to its language (only Go, only Python and so on) >>>>> i.e. use >>>>> > >>> the >>>>> > >>> > >>>> low-level API. >>>>> > >>> > >>>> >>>>> > >>> > >>>> Sorry for the confusion, >>>>> > >>> > >>>> Niklas >>>>> > >>> > >>>> >>>>> > >>> > >>>> >>>>> > >>> > >>>>> On 10 July 2014 15:55, Dominic Hamon < >>>>> [email protected] >>>>> > >>> <http://JAVASCRIPT-BLOCKED;>> wrote: >>>>> > >>> > >>>>> >>>>> > >>> > >>>>> In my dream world, we wouldn't need any native bindings. >>>>> I can >>>>> > >>> imagine >>>>> > >>> > >>>>> having example frameworks or starter frameworks that use >>>>> the >>>>> > >>> low-level >>>>> > >>> > >>>>> API >>>>> > >>> > >>>>> (the wire protocol with protocol buffers for message >>>>> passing), >>>>> > >>> but >>>>> > >>> > >>>>> nothing >>>>> > >>> > >>>>> like we have that needs C or JNI, etc. >>>>> > >>> > >>>>> >>>>> > >>> > >>>>> >>>>> > >>> > >>>>> >>>>> > >>> > >>>>> >>>>> > >>> > >>>>> On Thu, Jul 10, 2014 at 3:26 PM, Niklas Nielsen < >>>>> > >>> [email protected] <http://JAVASCRIPT-BLOCKED;>> >>>>> > >>> > >>>>> wrote: >>>>> > >>> > >>>>> >>>>> > >>> > >>>>>> Hi all, >>>>> > >>> > >>>>>> >>>>> > >>> > >>>>>> I wanted to start a discussion around the language >>>>> bindings in >>>>> > >>> the >>>>> > >>> > >>>>>> wild >>>>> > >>> > >>>>>> (Go, Haskell, native Python, Go, Java and so on) and >>>>> possibly >>>>> > >>> get to a >>>>> > >>> > >>>>>> strategy where we start bringing those into Mesos >>>>> proper. As >>>>> > >>> most >>>>> > >>> > >>>>>> things >>>>> > >>> > >>>>>> points towards, it will probably make sense to focus on >>>>> the >>>>> > >>> native >>>>> > >>> > >>>>>> "bindings" leveraging the low-level API. To name one >>>>> candidate >>>>> > >>> to >>>>> > >>> > >>>>>> start >>>>> > >>> > >>>>>> with, we are especially interested in getting Go native >>>>> support >>>>> > >>> in >>>>> > >>> > >>>>>> Mesos >>>>> > >>> > >>>>>> proper (and in a solid state). So Vladimir, we'd be >>>>> super >>>>> > >>> thrilled to >>>>> > >>> > >>>>> start >>>>> > >>> > >>>>>> collaborating with you on your current work. >>>>> > >>> > >>>>>> >>>>> > >>> > >>>>>> We are interested to hear what thoughts you all might >>>>> have on >>>>> > >>> this. >>>>> > >>> > >>>>>> >>>>> > >>> > >>>>>> Thanks, >>>>> > >>> > >>>>>> Niklas >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > -- >>>>> > >>> > Gu Yifan >>>>> > >>> > >>>>> > >>> > >>>>> > >>> >>>>> > >>> -- >>>>> > >>> Cheers, >>>>> > >>> Timothy St. Clair >>>>> > >>> Red Hat Inc. >>>>> > >>> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> > >> -- >>>>> > >> Cheers, >>>>> > >> Timothy St. Clair >>>>> > >> Red Hat Inc. >>>>> > >> >>>>> > > >>>>> > > >>>>> > > >>>>> > > -- >>>>> > > Vladimir Vivien >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > -- >>>>> > > Cheers, >>>>> > > Timothy St. Clair >>>>> > > Red Hat Inc. >>>>> > > >>>>> > >>>>> >>>> >>>> >>> >> >
