Hi BCS, On Fri, 07 May 2010 01:54:17 +0000, BCS wrote: n> > I haven't watched the video (not enough bandwidth) but I'd like a well > done package tool as well. Thinking in the direction of a spec, what do > you like about the Go system?
I'll summarize the video (which, by the way, contains everything I know about the Go package system!). The screencast shows a developer writing a short application with a reusable function in it. He factors the function out into a separate module in a subdirectory, then (with the aid of a copied Makefile) installs the new module into a shared location on the host, so it can be reused by other apps. Next, he pushes the new module into a github repository, and reinstalls the module from github like so: goinstall github.com/nf/mytools # (github user nf, package mytools) The source is checked out of github, compiled and installed in the central Go 'pkg' directory (both sources and objects are stored there). He commits some new changes to github, and reinstalls the updated module using the same goinstall command. Finally, he returns to his original application, and replaces the Go equivalent of "import mytools;" with "import github.com/nf/mytools;" and the application now refers to the version of mytools that was installed from Github. > Some things I'd like: > > -It must work offline (via sneeker net) and from trivial local static > mirrrors/archives -It should support executing bash commands, D > executables or something that does most of what people use that kind of > things for. (Some kind of sandbox/security feature might be needed) Stepping away from Go specifics, I like the idea of a system that does some of the following things. (just a brain-dump, no particular order): (by the way, I'm overloading the term "package" here, at times to mean a package in the D sense, but usually to mean a downloadable and installable unit.) * It knows how to build and install packages from an arbitrary source, specified by a URL: a git/hg/svn repository, a tarball on the filesystem, a zipfile on a Web server. Exotic sources/formats could possibly be added with plugins. * It keeps the ceremony of sharing code to a minimum. It should be easy to publish a simple library package, without reading an excess of documentation or writing an excess of package metadata. * It separates the concerns of installing packages and managing a central repository of packages. As an analogy, consider 'dpkg' vs. 'apt-get' on Debian systems: one knows about the mechanics of package installation and removal, the other can search a central catalogue, automatically install dependencies, etc. While 'd-get' would be very nice, 'd-dpkg' is the essential building block. This is important: the core package tool should not try to do more than absolutely necessary to simply fetch and install code. * It enables (but does not provide) apt-get-like use by providing ways to (optionally) declare package dependencies (platform, D-version, other needed packages), and include metadata about the package (author, website, etc.) but does not require a complex package description for simple packages. (An important usability question is, what is the minimum package description that should be required of a package's author? Can we work with no metadata at all?) * It should be easy to have both user and global package directories. It shouldn't be necessary to 'have root' to install a package. * It doesn't depend on Make or another build tool, at least not for basic library packages. (From the DSSS docs, it seems that DSSS was able to manage dependency graphs, and do the right thing with a simple 'dsss compile and install' command.) Having said that, a package with complex build requirements ought to be buildable with a Makefile, perhaps requiring a 'd-build' target in the Makefile that builds the package into a state that meets our package system's installation requirements. * It makes it easy for library consumers to integrate the code. After I've installed some third-party libaries, I'd like a tool that can manage (or support) the build of an application that uses those libraries -- for example, by figuring out which object files need to be linked in, based on the modules I've imported. This could be a tool that just calculates the necessary compiler and linker flags, and/or a tool that runs the compiler/linker for me with those flags in place, and/or a tool that builds a suitable Makefile for me, based on my project's dependencies. (Arguably, this feature is outside the scope of a simple fetch-and-install system; but if it's not easy to use those fetched packages, then the overall value of the system is weakened.) * Nice to have: a way to allow multiple versions of the same package to be installed locally, and a corresponding way to specify which version of the package my application is using. * Nice to have: a way to specify dependencies between third-party packages, and dependencies on external libraries (not necessarily a way to install those external dependencies, just a declaration). * Nice to have: some support for running unit tests, especially a basic 'pre-install' unit test to confirm that all dependencies are in place (if possible). * Nice to have: a way to publish to a central feed of "known D packages". Less formal than dsource.org: not a user site, but a machine-readable one, and possibly distributed/mirrorable. As an example, the Clojure community has a "clojars.org" site which lets anyone publish Clojure packages there. Clojars has very limited support for searching/browsing, it's really more of a key/value store: it is intended for use by automated tools (it's a Maven repository, as I understand it). Clojars actually maintains an archive of those packages (as jar files), this might/might not be a good idea. I think the "feed" is the key thing. * Centralizing the feed-of-packages would also enable another useful feature: a way of mapping a canonical package name (and optionally, version) to the URL of the source for that package. (Possibly multiple URLs for the same source.) This is headed into 'apt-get' territory, but minimizing the responsibility of the central repository to "here's where to find package x.y.z" without any other assurances (e.g., that x.y.z will build properly on your system). Conventions about package naming (should my DSP package be called "dsp", or "fawcett.dsp", "algorithms.signals.dsp", etc.) and package-name dispute resolution are orthogonal and should be left out of the scope of the system. These next thoughts are outside the scope; but a good design of the core tool won't get in the way of future progress in these areas: * Thinking ahead: ensure the system can be used by continuous integration systems, so that e.g. a buildbot could read the feed for new "known D packages", and attempt to install them on various D-version and OS permutations, reporting the results. (I'm not suggesting that the buildbot is part of the package management system, just that it should be easy for buildbot maintainers to be aware of new package versions so they can be tested.) The same facility could be used by developers who want to keep a "mirror" of some or all known D packages (even though those packages might reside upstream on several different hosts). * Thinking ahead: eventually we hope for a nice site where people can browse, review, grade, and comment upon packages. An opt-in system where consumers agree to publish their installation activity (like Debian's 'popularity contest'). * Thinking ahead: easy message-digests and digital signatures for packages, enabling a truly distributed package-publishing system. Anyone can host a d-libraries mirror, and a signed "central feed" document can publish the known hashes for those packages to avoid man-in-the-middle attacks. * Thinking ahead: I really like the way github.com shows a 52-week activity graph on projects that are hosted there. I don't think we can do exactly that, but it would be nice to be able to generate a report for all packages, showing how frequently they are updated and maintained. * I personally favour a system that works with D2 only, but I am new here. :) Otherwise, the implications of D1/D2 and Phobos/Tango need to be considered by more seasoned heads than mine. And for now, I'm all thought out. :) Your thoughts? Best, Graham
