Hi David and Jimmy, On Thu, 16 May 2013, Jimmy Kaplowitz wrote:
> All of these things will need to be a multidirectional discussion with > various participants having their areas of familiarity and expertise, > and other areas where listening to other perspectives and needs will > be most productive. My individual role is somewhat in the middle here, > trying to help bridge gaps as well as doing some of the technical > work. I'm very happy that my colleagues want to do the right thing for > both Google's customers and Debian. "do the right thing" is indeed a worthwhile motivator ;) I am all for it too, so please take my comments/clarifications bellow in that perspective: > > From Google's perspective, there are a few issues we don't yet know > > how to solve: > > 1) As our online services change (rapidly), it's important to get > > our customers new versions of our tools quickly and easily. It's > > terrible for customers to get an N-month-old-version of our tools in a > > Debian-6 or Debian-7 repo, and try to use that. Our current solution > > to that problem is to build and push images every month or so. We'd > > *love* a mechanism wherein we can push new versions of our tools to > > official debian archives every month or two... none of the tools in question is in any official Debian stable release (e.g. nor in 6 AKA oldstable/squeeze, neither in 7 AKA stable/wheezy), and cannot be ever uploaded there -- that train is gone. So for the next 1-3 years, until next stable Debian release comes out you should not worry that users of official Debian repositories would get some stale version. Packages could be uploaded to Debian unstable (or experimental) only and nearly as often as you like (and your Debian "sponsor", e.g. Jimmy would be capable to allocate time for) -- e.g. could be many times a day and the official archive is updated twice a day I believe. Such packages could even be forbidden from entering testing, to which packages migrate usually after 10 days (or could even be shorter) if no grave unfixed bugs present, in case you really want to have a short leash there (but I do not think it is necessary). So altogether I really do not see ANY problem with rapid development of your tools and their availability in Debian archives. Moreover presence in the archives with automatic ways to update (instead of "go to website and download a new version") sounds like the only sensible approach here. The only way I see for older versions being dumped upon users within upcoming 1-2 years would be when Ubuntu (or some other smaller derivative) picks up such a package from Debian and includes in their release... but I guess we could deal with that one way (keep them in experimental) or another (just talk and express the reason not to include into non-rolling releases). > > 2) gsutil has an array of c++ and python dependencies that are > > available in Debian-6, but not with high-enough versions of those > > dependencies. once again: we would not even be able to upload to the official Debian 6, nor Debian 7 -- those are released already, nothing could be added (besides to backports. repository). And if versioned dependencies are "good enough" in Debian sid ATM -- there is no problem. If they are outdated -- please say so, we will work to fix that ;) > > We have a binary-only debian package that installs all > > the stuff necessary here, but it's unclear to us whether this is "good > > enough" for inclusion in official Debian repos. Our .tar.gz version > > takes these dependencies and includes them essentially "statically > > linked" in the gsutil installation directory, which is a little > > unconventional as well. yes -- it would not work for 'official' Debian binary packages -- but that is not necessary (as per above). As for backport builds for older releases there could be multiple ways to resolve the conundrum, and in any case you might end up creating your own APT repository to provide those binary packages built for different releases. That is similar to what we are doing in NeuroDebian (http://neuro.debian.net) where we do provide backport builds of all packages we also upload to Debian proper. And we strive to not waste effort in maintaining two copies of the same package. Quite often, indeed system-provided libraries are outdated and it would have been detrimental to the integrity of the underlying distribution users use if we simply provided "fresh" builds of a new library thus replacing some older but stable version. So far we adhered to two approaches: 1. E.g. cmtk package & mxml-- as long licenses permit I do not strip needed 3rd party libraries in source distribution -- http://anonscm.debian.org/gitweb/?p=pkg-exppsy/cmtk.git;a=tree;hb=HEAD Utilities/mxml ships sources (not binaries) which are built/linked against ONLY on those systems lacking up-to-date version. But within Debian (i.e. while uploading to sid) I assure that it builds/uses system-wide installed/maintained versions, so those sources are not used. This allows us to build cmtk across wide range of Debian and Ubuntu releases built from a single source package while still fulfilling Debian policy: http://neuro.debian.net/pkgs/cmtk.html . So, pretty much, you could stay with what you are doing and probably shipping those 3rd party sources inside (or only for backported packages), or #2: 2. E.g. psychtoolbox-3 package & glew 1.9 -- we have glew 1.9 only in Debian experimental... So I have backported glew as of 1.9 and provided it from neurodebian as versioned binary packages (i.e. libglew1.9 and libglew1.9-dev). libglew1.9-dev conflicts with original libglew-dev but we do not need to have them both present on the same system while building psychtoolbox-3 and libglew1.9 natively coexists with libglew1.7. When 1.9 comes to proper Debian/Ubuntu releases those versions would superseed my backported ones and everyone should stay happy. So -- now I still build psychtoolbox-3 from the same sources for Debian proper and NeuroDebian backports: http://neuro.debian.net/pkgs/octave-psychtoolbox-3.html ... so the point is -- if you would be needing backport builds -- it should be possible while keeping packaging still acceptable for Debian. > > 3) Really, focusing on gsutil and gcutil are a little narrow-minded. > > What we at Google *really* want, is a good understanding of the > > process and ideal mechanism to cook those artifacts that integrates > > with Google build tools and with Debian's release tools. Then we can > > standardize all our Google properties and use this mechanism, and make > > Debian and Google "tier one" partners for releasing software. Sounds like a sound goal, but from my "user" point of view -- "I do not care" ;) I am just told by you that I need to use gcutil to use GCE -- but it is not accessible for me on Debian (click/download/adjust PATH is not "accessible" in my terms). And that is what I am trying to address here. And hopefully by the time of next stable Debian things become more clear to aim for the holy mighty artifacts ;) > > 4) More deeply, beyond just those tools, there are several bugs > > we've found in Debian OS, which lead to highly degraded network > > performance, and what looks to be massive CPU starvation in cloud > > environments. We want to fix this problem in the Debian-6 and > > Debian-7 images we provide but the process to do that is still unclear > > to us. Let us know if you need any help/guidance -- I would first talk to corresponding Debian folks (e.g. kernel team). But this is a side topic from the 1-2 (shipping tools you currently ask users to use). > > Really, our goal here is to do The Right Thing. The problem is that > > from our perspective, it's a discussion that is going to take some > > time. But we're eager to have that discussion. Here you go -- we could discuss... or may be we could simply start cooking ;) Do you see any particular (technical) problem with gcutil source distribution and its goodness of fit for distribution in Debian sid (i.e. always rolling, uploads could be multiple times a day etc)? -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik -- To UNSUBSCRIBE, email to [email protected] with a subject of "unsubscribe". Trouble? Contact [email protected] Archive: http://lists.debian.org/[email protected]
