On May 24, 2011, at 1:17 PM, David Golden wrote: > Then, given one of those three ordered list of tarballs that satisfy > all prereqs, it should be possibly to repeatably deploy an application > with a known set of module versions, even as the "latest" on CPAN > evolves.
. Wow! This is getting juicy. It's easy for me to get carried away with all the intricacies of dependency management. But for my task at hand, I think it might be the wrong problem. So for the sake of this discussion, let's put aside the issue of how a private CPAN (really, the 02packages) could be constructed to have a magical mixture of modules. Instead, I want to think about how a private CPAN mirror fits into the development cycle. This might be easiest by walking through a hypothetical scenario... Let's suppose I wan to write a new application in Perl. By "application", I really just mean some set of modules and scripts. And let's assume that my development, testing, and deployment environments all have a perfectly clean version of perl -- no modules have been added or updated from the core. I want my app to leverage the CPAN tool chain, so I setup a CPAN-style distribution for my source code. It has lib/, bin/, and t/ directories, and all the usual trappings. I'm also going to use Module::Build. And of course, all this is in some source control system (assume svn for now). After a couple of Mountain-Dews, I have some code and tests and everything passes. Commit. Now I want to add some feature that requires a module from CPAN. So I go shopping on search.cpan.org and find one I like called Frobulator::Simple. At this point, I might go ask my sysadmin to install Frobulator::Simple (and all its dependencies) in /usr/lib/perl5 on the development machine. Or if I'm smart, I might use local::lib to install it somewhere in my home directory. But these both suck, because changing /usr/lib/perl5 will affect everyone else, and who knows what else I might have installed in my home directory. So instead, let's say I add Frobulator::Simple to the "requires" list (or "build_requires", or whatever) in my Build.PL. Now I run the "installdeps" target, which fetches the latest Frobulator::Simple and all its dependencies from a public CPAN and installs them. Now this is where it gets interesting. Rather than install in site_perl, it installs into a directory right inside my project (let's call it "dlib" for "dependent libraries"). At the same time, it stashes the tar balls in a directory structure suitable for a CPAN mirror, which is also right inside my project directory (let's call it "lpan", for "local perl archive network"). Finally, it generates the 02packages for the lpan/. Everything in lpan/ will be placed under revision control too. Now I have all my application code and my dependencies in one local, revision-controlled location. When I run the "build" target, all my modules are copied into the blib/ directory as usual. And when I run the "test" target, both blib/ and dlib/ are placed on @INC. If the tests pass, then commit. If not, then fix my code. Or I can "svn revert" my lpan/ directory, change the "requires" list in Build.PL to try a new module, and run the "installdeps" target again. Rinse and repeat as needed. This goes on for a while as my application evolves until I have several dependencies. Dependencies that I want to keep are in the growing dpan/ directory and committed to svn. Meanwhile, the dlib directory is clouding up with all the modules I've experimented with. Periodically, I want to test the whole stack. So now I run the "realclean" target, which wipes out dlib/. Next, I run an "installdepslocal" target, which basically does an "installdeps", but this time it pulls everything from the lpan. This effectively flushes out any modules that fail since, over time, I may have upgraded some of their dependencies in my lpan/ and those upgrades may not be backward compatible. If some of the dependencies do fail, it is my job to figure out what to do. This might mean removing a newer distro from the lpan/ in favor of an older one. Or maybe I'll make a patched version of the distro and put it in the lpan/. Or perhaps I'll just switch to a different dependency altogether. How this all gets done is out of scope for this discussion, but I imagine that we could generate some kind of map that tracks all the dependencies for the app. This could help developers see the interdependencies and figure out how an upgrade or downgrade will impact the app. You could also have special build targets that assist with adding, removing, or upgrading specific sub-trees in the dependency graph. But for now, let's assume I've got a working stack.. When I run the "test" target again, I'm now testing my code against a pure set of dependencies. The traditional "disttest" target might also be enhanced to first run "installdepslocal" in the dist directory. Once my application code is golden, I run a "release" target that builds the tar.gz of my application and shoves into the lpan/. Again, the 02packages is automatically updated. Commit. Now let's say we use the CPAN tool chain for deployment too. My application and all it's dependencies are in the lpan/, so I can just login to the target host, point to the lpan in SVN, and say "cpan My::App". Again, we'd use local::lib to ensure everything does into an app-specific location. This is especially easy with SVN, since all the tarballs in the lpan can be accessed with HTTP requests. Or if we prefer, we can build the local lib/ directory on a separate host, archive it up, and ship it to the target that way. So far, I've described all this in terms of Module::Build targets, but some (perhaps all) of this could be done with Dist::Zilla plugins. One thing I like about this model is that it works well for team development, since all the dependencies converge in the lpan in svn. And since lpan is part of the project, it plays nicely with branching and merging. Also, I like it because it requires absolutely no administration -- each app has its own lpan/, which is totally controlled by the developers. Lastly, it doesn't require rejiggering the entire CPAN tool chain and/or backfilling CPAN distros with some new metadata. I realize this doesn't solve the "big" problem of automatically building a particular CPAN mirror with a magical mix modules. But based on my own development patterns, I think this model could provide a workable framework for organically evolving application dependencies, while providing a stable and reproducible dependency stack at the same time. What do you think? Jeffrey Thalhammer Imaginative Software Systems vcard: http://www.imaginative-software.com/contact/jeff.vcf
