That said, the bulk of Google Cloud just wants to ship completely 100% opensource code here. It shouldn't be terrible to get it right, and as a posh + proper citizen of jessie. But I'm afraid I'm not sure I see the path completely yet. Especially on the "how to build the packages for these things, when versionitis starts setting in" front.
-david On Thu, May 16, 2013 at 12:42 PM, David McWherter <[email protected]> wrote: > I think keeping us in unstable and backports is totally reasonable for > the time being. Especially if that helps us with some of the > binary-only packaging we may need to deal with. Jimmy indicated there > may be quality-control issues with .debs produced by existing Google > tools, and by FPM, that might still make that hard to deal with, but > that could be fixed in time too. > > I realize that from your user point of view, non-deb is > non-accessible. And I want to solve that problem too, but the simple > fact is that 90% of Google Compute customer base doesn't have that > problem. The biggest issue for me, and for them, I think, is that > 'apt-get upgrade gcutil' doesn't work for them. Which would also > solve your problem, I think :) > > I think packaging 'gcutil' as a debian that installs cleanly would be > relatively easy, and it's on our plate (of course we've all been > overworked with other things getting ready for Google IO). I think a > "binary" deb is most useful for users. I'm actually not sure what > differentiates a "binary" deb from a "source" deb if the tool is 100% > python. Our gcutil would statically link in a bunch of other packages > it needs. 'gsutil' i think is still a harder story. We're not 100% > confident that the .deb that we create works correctly on both debian6 > and debian7 because of the crazy version mismatchy problems. > > -david > > On Thu, May 16, 2013 at 12:08 PM, Yaroslav Halchenko > <[email protected]> wrote: >> Hi David and Jimmy, >> >> On Thu, 16 May 2013, Jimmy Kaplowitz wrote: >> >>> All of these things will need to be a multidirectional discussion with >>> various participants having their areas of familiarity and expertise, >>> and other areas where listening to other perspectives and needs will >>> be most productive. My individual role is somewhat in the middle here, >>> trying to help bridge gaps as well as doing some of the technical >>> work. I'm very happy that my colleagues want to do the right thing for >>> both Google's customers and Debian. >> >> "do the right thing" is indeed a worthwhile motivator ;) I am all for >> it too, so please take my comments/clarifications bellow in that perspective: >> >>> > From Google's perspective, there are a few issues we don't yet know >>> > how to solve: >> >>> > 1) As our online services change (rapidly), it's important to get >>> > our customers new versions of our tools quickly and easily. It's >>> > terrible for customers to get an N-month-old-version of our tools in a >>> > Debian-6 or Debian-7 repo, and try to use that. Our current solution >>> > to that problem is to build and push images every month or so. We'd >>> > *love* a mechanism wherein we can push new versions of our tools to >>> > official debian archives every month or two... >> >> none of the tools in question is in any official Debian stable release >> (e.g. nor in 6 AKA oldstable/squeeze, neither in 7 AKA >> stable/wheezy), and cannot be ever uploaded there -- that train is gone. >> >> So for the next 1-3 years, until next stable Debian release comes out >> you should not worry that users of official Debian repositories would >> get some stale version. Packages could be uploaded to Debian unstable >> (or experimental) only and nearly as often as you like (and your Debian >> "sponsor", e.g. Jimmy would be capable to allocate time for) -- e.g. >> could be many times a day and the official archive is updated twice a >> day I believe. Such packages could even be forbidden from >> entering testing, to which packages migrate usually after 10 days (or >> could even be shorter) if no grave unfixed bugs present, in case you >> really want to have a short leash there (but I do not think it is >> necessary). >> >> So altogether I really do not see ANY problem with rapid development of >> your tools and their availability in Debian archives. Moreover presence >> in the archives with automatic ways to update (instead of "go to website >> and download a new version") sounds like the only sensible >> approach here. >> >> The only way I see for older versions being dumped upon users within >> upcoming 1-2 years would be when Ubuntu (or some other smaller >> derivative) picks up such a package from Debian and includes in >> their release... but I guess we could deal with that one way (keep them >> in experimental) or another (just talk and express the reason not to >> include into non-rolling releases). >> >> >>> > 2) gsutil has an array of c++ and python dependencies that are >>> > available in Debian-6, but not with high-enough versions of those >>> > dependencies. >> >> once again: we would not even be able to upload to the official >> Debian 6, nor Debian 7 -- those are released already, nothing could be >> added (besides to backports. repository). And if versioned dependencies >> are "good enough" in Debian sid ATM -- there is no problem. If they >> are outdated -- please say so, we will work to fix that ;) >> >>> > We have a binary-only debian package that installs all >>> > the stuff necessary here, but it's unclear to us whether this is "good >>> > enough" for inclusion in official Debian repos. Our .tar.gz version >>> > takes these dependencies and includes them essentially "statically >>> > linked" in the gsutil installation directory, which is a little >>> > unconventional as well. >> >> yes -- it would not work for 'official' Debian binary packages -- >> but that is not necessary (as per above). >> >> As for backport builds for older releases there could be multiple ways >> to resolve the conundrum, and in any case you might end up creating your >> own APT repository to provide those binary packages built for different >> releases. That is similar to what we are doing in NeuroDebian >> (http://neuro.debian.net) where we do provide backport builds of all >> packages we also upload to Debian proper. And we strive to not waste >> effort in maintaining two copies of the same package. Quite often, >> indeed system-provided libraries are outdated and it would have been >> detrimental to the integrity of the underlying distribution users use if >> we simply provided "fresh" builds of a new library thus replacing some >> older but stable version. So far we adhered to two approaches: >> >> 1. E.g. cmtk package & mxml-- as long licenses permit I do not strip >> needed 3rd party libraries in source distribution -- >> http://anonscm.debian.org/gitweb/?p=pkg-exppsy/cmtk.git;a=tree;hb=HEAD >> Utilities/mxml ships sources (not binaries) which are built/linked >> against ONLY on those systems lacking up-to-date version. But within >> Debian (i.e. while uploading to sid) I assure that it builds/uses >> system-wide installed/maintained versions, so those sources are not >> used. This allows us to build cmtk across wide range of Debian and >> Ubuntu releases built from a single source package while still >> fulfilling Debian policy: >> http://neuro.debian.net/pkgs/cmtk.html . So, pretty much, you could >> stay with what you are doing and probably shipping those 3rd party >> sources inside (or only for backported packages), or #2: >> >> 2. E.g. psychtoolbox-3 package & glew 1.9 -- we have glew 1.9 only in >> Debian experimental... So I have backported glew as of 1.9 and provided >> it from neurodebian as versioned binary packages (i.e. libglew1.9 and >> libglew1.9-dev). libglew1.9-dev conflicts with original libglew-dev but >> we do not need to have them both present on the same system while >> building psychtoolbox-3 and libglew1.9 natively coexists with >> libglew1.7. When 1.9 comes to proper Debian/Ubuntu releases those >> versions would superseed my backported ones and everyone should stay >> happy. So -- now I still build psychtoolbox-3 from the same sources for >> Debian proper and NeuroDebian backports: >> http://neuro.debian.net/pkgs/octave-psychtoolbox-3.html >> >> ... so the point is -- if you would be needing backport builds -- it >> should be possible while keeping packaging still acceptable for Debian. >> >>> > 3) Really, focusing on gsutil and gcutil are a little narrow-minded. >>> > What we at Google *really* want, is a good understanding of the >>> > process and ideal mechanism to cook those artifacts that integrates >>> > with Google build tools and with Debian's release tools. Then we can >>> > standardize all our Google properties and use this mechanism, and make >>> > Debian and Google "tier one" partners for releasing software. >> >> Sounds like a sound goal, but from my "user" point of view -- "I do not >> care" ;) I am just told by you that I need to use gcutil to use GCE -- >> but it is not accessible for me on Debian (click/download/adjust PATH is >> not "accessible" in my terms). And that is what I am trying to >> address here. And hopefully by the time of next stable Debian things >> become more clear to aim for the holy mighty artifacts ;) >> >>> > 4) More deeply, beyond just those tools, there are several bugs >>> > we've found in Debian OS, which lead to highly degraded network >>> > performance, and what looks to be massive CPU starvation in cloud >>> > environments. We want to fix this problem in the Debian-6 and >>> > Debian-7 images we provide but the process to do that is still unclear >>> > to us. >> >> Let us know if you need any help/guidance -- I would first talk to >> corresponding Debian folks (e.g. kernel team). But this is a side topic >> from the 1-2 (shipping tools you currently ask users to use). >> >>> > Really, our goal here is to do The Right Thing. The problem is that >>> > from our perspective, it's a discussion that is going to take some >>> > time. But we're eager to have that discussion. >> >> Here you go -- we could discuss... or may be we could simply start >> cooking ;) Do you see any particular (technical) problem with >> gcutil source distribution and its goodness of fit for distribution in >> Debian sid (i.e. always rolling, uploads could be multiple times a day >> etc)? >> >> -- >> Yaroslav O. Halchenko, Ph.D. >> http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org >> Senior Research Associate, Psychological and Brain Sciences Dept. >> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 >> Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 >> WWW: http://www.linkedin.com/in/yarik -- To UNSUBSCRIBE, email to [email protected] with a subject of "unsubscribe". Trouble? Contact [email protected] Archive: http://lists.debian.org/cag0yj4jwh4nx_z8njow+anbgua7rubux95n3kggu2pgqeuj...@mail.gmail.com
