I think keeping us in unstable and backports is totally reasonable for the time being. Especially if that helps us with some of the binary-only packaging we may need to deal with. Jimmy indicated there may be quality-control issues with .debs produced by existing Google tools, and by FPM, that might still make that hard to deal with, but that could be fixed in time too.
I realize that from your user point of view, non-deb is non-accessible. And I want to solve that problem too, but the simple fact is that 90% of Google Compute customer base doesn't have that problem. The biggest issue for me, and for them, I think, is that 'apt-get upgrade gcutil' doesn't work for them. Which would also solve your problem, I think :) I think packaging 'gcutil' as a debian that installs cleanly would be relatively easy, and it's on our plate (of course we've all been overworked with other things getting ready for Google IO). I think a "binary" deb is most useful for users. I'm actually not sure what differentiates a "binary" deb from a "source" deb if the tool is 100% python. Our gcutil would statically link in a bunch of other packages it needs. 'gsutil' i think is still a harder story. We're not 100% confident that the .deb that we create works correctly on both debian6 and debian7 because of the crazy version mismatchy problems. -david On Thu, May 16, 2013 at 12:08 PM, Yaroslav Halchenko <[email protected]> wrote: > Hi David and Jimmy, > > On Thu, 16 May 2013, Jimmy Kaplowitz wrote: > >> All of these things will need to be a multidirectional discussion with >> various participants having their areas of familiarity and expertise, >> and other areas where listening to other perspectives and needs will >> be most productive. My individual role is somewhat in the middle here, >> trying to help bridge gaps as well as doing some of the technical >> work. I'm very happy that my colleagues want to do the right thing for >> both Google's customers and Debian. > > "do the right thing" is indeed a worthwhile motivator ;) I am all for > it too, so please take my comments/clarifications bellow in that perspective: > >> > From Google's perspective, there are a few issues we don't yet know >> > how to solve: > >> > 1) As our online services change (rapidly), it's important to get >> > our customers new versions of our tools quickly and easily. It's >> > terrible for customers to get an N-month-old-version of our tools in a >> > Debian-6 or Debian-7 repo, and try to use that. Our current solution >> > to that problem is to build and push images every month or so. We'd >> > *love* a mechanism wherein we can push new versions of our tools to >> > official debian archives every month or two... > > none of the tools in question is in any official Debian stable release > (e.g. nor in 6 AKA oldstable/squeeze, neither in 7 AKA > stable/wheezy), and cannot be ever uploaded there -- that train is gone. > > So for the next 1-3 years, until next stable Debian release comes out > you should not worry that users of official Debian repositories would > get some stale version. Packages could be uploaded to Debian unstable > (or experimental) only and nearly as often as you like (and your Debian > "sponsor", e.g. Jimmy would be capable to allocate time for) -- e.g. > could be many times a day and the official archive is updated twice a > day I believe. Such packages could even be forbidden from > entering testing, to which packages migrate usually after 10 days (or > could even be shorter) if no grave unfixed bugs present, in case you > really want to have a short leash there (but I do not think it is > necessary). > > So altogether I really do not see ANY problem with rapid development of > your tools and their availability in Debian archives. Moreover presence > in the archives with automatic ways to update (instead of "go to website > and download a new version") sounds like the only sensible > approach here. > > The only way I see for older versions being dumped upon users within > upcoming 1-2 years would be when Ubuntu (or some other smaller > derivative) picks up such a package from Debian and includes in > their release... but I guess we could deal with that one way (keep them > in experimental) or another (just talk and express the reason not to > include into non-rolling releases). > > >> > 2) gsutil has an array of c++ and python dependencies that are >> > available in Debian-6, but not with high-enough versions of those >> > dependencies. > > once again: we would not even be able to upload to the official > Debian 6, nor Debian 7 -- those are released already, nothing could be > added (besides to backports. repository). And if versioned dependencies > are "good enough" in Debian sid ATM -- there is no problem. If they > are outdated -- please say so, we will work to fix that ;) > >> > We have a binary-only debian package that installs all >> > the stuff necessary here, but it's unclear to us whether this is "good >> > enough" for inclusion in official Debian repos. Our .tar.gz version >> > takes these dependencies and includes them essentially "statically >> > linked" in the gsutil installation directory, which is a little >> > unconventional as well. > > yes -- it would not work for 'official' Debian binary packages -- > but that is not necessary (as per above). > > As for backport builds for older releases there could be multiple ways > to resolve the conundrum, and in any case you might end up creating your > own APT repository to provide those binary packages built for different > releases. That is similar to what we are doing in NeuroDebian > (http://neuro.debian.net) where we do provide backport builds of all > packages we also upload to Debian proper. And we strive to not waste > effort in maintaining two copies of the same package. Quite often, > indeed system-provided libraries are outdated and it would have been > detrimental to the integrity of the underlying distribution users use if > we simply provided "fresh" builds of a new library thus replacing some > older but stable version. So far we adhered to two approaches: > > 1. E.g. cmtk package & mxml-- as long licenses permit I do not strip > needed 3rd party libraries in source distribution -- > http://anonscm.debian.org/gitweb/?p=pkg-exppsy/cmtk.git;a=tree;hb=HEAD > Utilities/mxml ships sources (not binaries) which are built/linked > against ONLY on those systems lacking up-to-date version. But within > Debian (i.e. while uploading to sid) I assure that it builds/uses > system-wide installed/maintained versions, so those sources are not > used. This allows us to build cmtk across wide range of Debian and > Ubuntu releases built from a single source package while still > fulfilling Debian policy: > http://neuro.debian.net/pkgs/cmtk.html . So, pretty much, you could > stay with what you are doing and probably shipping those 3rd party > sources inside (or only for backported packages), or #2: > > 2. E.g. psychtoolbox-3 package & glew 1.9 -- we have glew 1.9 only in > Debian experimental... So I have backported glew as of 1.9 and provided > it from neurodebian as versioned binary packages (i.e. libglew1.9 and > libglew1.9-dev). libglew1.9-dev conflicts with original libglew-dev but > we do not need to have them both present on the same system while > building psychtoolbox-3 and libglew1.9 natively coexists with > libglew1.7. When 1.9 comes to proper Debian/Ubuntu releases those > versions would superseed my backported ones and everyone should stay > happy. So -- now I still build psychtoolbox-3 from the same sources for > Debian proper and NeuroDebian backports: > http://neuro.debian.net/pkgs/octave-psychtoolbox-3.html > > ... so the point is -- if you would be needing backport builds -- it > should be possible while keeping packaging still acceptable for Debian. > >> > 3) Really, focusing on gsutil and gcutil are a little narrow-minded. >> > What we at Google *really* want, is a good understanding of the >> > process and ideal mechanism to cook those artifacts that integrates >> > with Google build tools and with Debian's release tools. Then we can >> > standardize all our Google properties and use this mechanism, and make >> > Debian and Google "tier one" partners for releasing software. > > Sounds like a sound goal, but from my "user" point of view -- "I do not > care" ;) I am just told by you that I need to use gcutil to use GCE -- > but it is not accessible for me on Debian (click/download/adjust PATH is > not "accessible" in my terms). And that is what I am trying to > address here. And hopefully by the time of next stable Debian things > become more clear to aim for the holy mighty artifacts ;) > >> > 4) More deeply, beyond just those tools, there are several bugs >> > we've found in Debian OS, which lead to highly degraded network >> > performance, and what looks to be massive CPU starvation in cloud >> > environments. We want to fix this problem in the Debian-6 and >> > Debian-7 images we provide but the process to do that is still unclear >> > to us. > > Let us know if you need any help/guidance -- I would first talk to > corresponding Debian folks (e.g. kernel team). But this is a side topic > from the 1-2 (shipping tools you currently ask users to use). > >> > Really, our goal here is to do The Right Thing. The problem is that >> > from our perspective, it's a discussion that is going to take some >> > time. But we're eager to have that discussion. > > Here you go -- we could discuss... or may be we could simply start > cooking ;) Do you see any particular (technical) problem with > gcutil source distribution and its goodness of fit for distribution in > Debian sid (i.e. always rolling, uploads could be multiple times a day > etc)? > > -- > Yaroslav O. Halchenko, Ph.D. > http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org > Senior Research Associate, Psychological and Brain Sciences Dept. > Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 > Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 > WWW: http://www.linkedin.com/in/yarik -- To UNSUBSCRIBE, email to [email protected] with a subject of "unsubscribe". Trouble? Contact [email protected] Archive: http://lists.debian.org/CAG0yJ4KBcE7=hcndruuofszbqxdssvousdub0+2ngfx8ckh...@mail.gmail.com
