We have a lot of serious problems because we lack a database of installed
distributions, releases and files. There are serious problems with
implementing one given A) the limitations of the standard Perl install and B)
wedging it into existing systems. But I think I have a solution. Its similar
to how meta data was slipped into the ecosystem without requiring authors to
rewrite their releases or install a bunch of extra modules. It just happens
as part of the normal CPAN module upgrade process.
I've been thinking that a minimal package database could be created by putting
some hooks into ExtUtils::Install::install(), which every Perl build system
ultimately uses, to record what gets installed. That way when
ExtUtils::Install is upgraded, the user gets a build database without
upgrading everything else.
This would be a fairly straight forward process at install time...
1) Copy everything to a temp directory
2) Record everything in that temp directory
3) Copy everything from temp into the real location
You could probably optimize this by skipping the copy to temp and just have
install() record stuff as it goes by, but this is the dumb, simple, robust way
to do it.
Storage is a problem. The only reliable "database" Perl ships with is DBM, an
on disk hash, so we can't get too fancy. It might take several DBM files, but
this is enough to record information and do simple queries. What are those
queries?
* What version of the database is this?
* What distributions are installed?
* What release of a distribution is installed?
* What files are in that release?
* What version is that release?
* What location was a release installed into? (core, vendor, site, custom)
* What are the checksums of those files?
And the basic operations we need to support.
* Add a release (ie. install).
* Delete a release (and its files).
* Delete an older version of a release (as part of install).
* Delete an older version of a release, only if its in the same release
location. This is so CPAN installs don't delete vendor installed modules.
* Verify the files of a release.
* List distributions/releases installed.
It would also store the MYMETA data which gives us a lot of information (such
as dependencies) for free.
This is all totally doable, and efficient enough, with a small pile of DBM
files and Storable. Where to put the database is a bit more complicated, see
the list of open problems below.
There's lots and lots and lots of additional information which could be stored
and queries and operations to allow, but if we can get the basics working
it'll allow a heap of new solutions. And I think this is a SMOP.
Future possibilities include...
* Auto-upgrade to SQLite if ExtUtils::Install::DB::SQLite is installed.
If a special module is installed we can offer SQLite support (or whatever) for
a more advanced database. At install time it would copy the existing DBM
system into its own database.
In general, more functionality can be added as more optional (or bundled)
dependencies are available to the system. Through it all the basic DBM
database would continue to be redundantly maintained to provide a fallback
should those optional modules break or go away.
* Extra hooks into the install system.
ExtUtils::Install is sort of a black box. If it started to do more than just
copy files it would need a more interesting API. Rather than trying to cram
more options into install() it would be worthwhile to write a new API. Build
systems can check for the existence of the new API and use that if available
and do more interesting things with the database. This would be necessary to
support uninstall.
Problems include...
* Anything installed before the new ExtUtils::Install is lost.
Just have to live with that. It will slowly go away as the new
ExtUtils::Install gets into core, Perl is released and vendor Perls update
their Perl or core modules. It'll take time, but we're in the long run here.
* Anything installed outside the normal blib process is lost.
Initially, this is acceptable loses.
Ideally the install process would be expanded to better deal with things which
are not Perl libraries or programs. Build systems which have their own
methods of installing these things could add them directly using the install
database API. A lot of hand waving here.
* Upgrading the database.
I'd like to put some thought into how things are laid out initially to avoid a
lot of major revisions, and thought into what information should be recorded
so its available later, but eventually we're going to want to change the
"schema", such as it is with DBM files.
I figure this can happen as part of upgrading ExtUtils::Install. It checks
what version of the database you have and performs the necessary transforms to
bring it up to the current version. We know how to do this, just have to keep
it in mind and remember to implement it.
* Where to put the database? What about non-standard install locations?
$Config{archlib} would seem the obvious location, but it presents a
permissions problem. If a non-root user installs into their home
directory, you don't want them needing root to write to the installation
database. There's several ways to deal with this.
One is to simply not record non-standard install locations, but this loses
data and punishes all those local::lib users out there.
Another is to have a separate install database for non-standard install
locations. This makes sense to me, but it brings in the sticky problem
of having to merge install databases. Sticky, but still a SMOP. Once you
have to implement merging anyway, it now makes sense to have an install
database for each install location. One for core. One for vendor. One for
perl. And one for each custom location. This has a lot of advantages to
better fit how Perl layers module installs.
* allows separation of permissions
* allows queries of what's installed based on what's in @INC
That second one is important. When a normal user queries the database, they
want to get what's installed in the standard library location. When a
local::lib user queries the database, they want to get what's installed in the
standard library locations AND their own local lib.
In summary...
Not perfect, but gets us off the ground. Its not a great database, but it
does the important job of recording the critical install-time data for later
use. Its implementable within the current system. It doesn't require a bunch
of dependencies, just one upgrade. It works with most existing module
releases. It solves a major design problem with the Perl module system.
I think it's a Simple(?!) Matter Of Programming in ExtUtils::Install to get it
off the ground. IMO the most important bit of coordination is putting some
thought into what the basic database should look like so we don't have to
worry about complicated upgrades later.
Thoughts?
