Daniel, Thanks for writing! Indeed, most of the deps are OpenStack. The ones that aren't are basically gevent, and but I think others have already touched on the long-term goal of using multiprocessing as an alternative.
So far as limiting backend deps, I'd agree that wal-e-s3, etc. packages are probably the way to go, though it would take a little care to make those packages work out of the same repo. As for python-daemon, your reckoning may differ, but my own list of preferences would be: 1. If you don't have the requirements in http://legacy.python.org/dev/peps/pep-3143/#correct-daemon-behaviour, then just use subprocess to farm out the fetches. My own reckoning, though, is that you must need them or else you wouldn't have gone to the trouble. (Maybe to prevent shared network FD's, maybe to prevent polluting stdout / stderr; the rationale is fairly clear, but I'm either too casual or slow a reader to say.) 2. If you do require everything that is "daemonization", and you're willing to maintain it yourself, Alex Martelli's daemonization example <http://stackoverflow.com/questions/972362/spawning-process-from-python> is pretty straightforward and where I usually end up. On the one hand, it is sad that this stuff never has made it into the standard library. On the other, daemonization has just enough differences of opinion that the "one way to do it" may never make it in, notwithstanding . (For a hint of all that, see the in-depth comments and example Alex refers to at http://code.activestate.com/recipes/278731/. Mr. Finney also has quite a good discussion in his PEP from 2009.) In the rare cases where I had to do such a thing, I've just worked off of Alex's example, taking the parts I needed to take. 3. If you still require daemonization and don't want to write it yourself, daemonize <https://pypi.python.org/pypi/daemonize> seems to be a fairly similar implementation that lacks dependencies. . 4. Else, you could talk to Ben Finney about altering his install_requires and maybe removing the lockfile dependency. Well, sorry for the long story there. Please let me know which of those you find amenable. None of them are particularly hard, and I'm happy to do a little legwork on any of them . -HJB On Mon, Oct 20, 2014 at 5:10 PM, Daniel Farina <[email protected]> wrote: > On Mon, Oct 20, 2014 at 4:51 PM, <[email protected]> wrote: > > Hi, > > > > I've been using WAL-E for a few months for backups and recovery. It's > > great! > > > > WAL-E does, however, pull in 20 Python dependencies.* For an > infrastructure > > tool, > > that's almost an unmanageably large number. > > Regrettably a ton of these are from OpenStack. As it would turn out, > WAL-E does enough lazy-loading that not having all the dependencies > for all backends is not a problem. > > One person suggested making all backends optional, but I deep-sixed > this idea on the basis that, unrefined, it means that the > default-installation of WAL-E would be exactly useless. > > A better suggestion that would require some work is to have other > packages like "wal-e-{s3,openstack,azure}" that would do the obvious > and not install the other junk. > > > And thus it was a bit of a disappointment to see the new release > > candidate (thanks for WALE_S3_ENDPOINT, btw!) introduce three more > > dependencies through the use of python-daemon: > > > > python-daemon>=1.5.2 > > setuptools (through python-daemon) > > lockfile>=0.7 (through python-daemon) > > > > The dependency on setuptools is particularly pernicious, since pretty > > much all our virtualenvs are using distribute. > > I think it'd be reasonable to take stewardship of the daemonization > parts to avoid this, yes. Consider that I decided to own subprocess > because of unbackpatched bug fixes :/ > > > It seems like WAL-E only uses python-daemon in one place during > > operator.backup.wal_restore(). Is there any chance that could be > > refactored to use one of the various vendored daemon implementations? > > Yes. Do you have a recommendation of one that works very reliably? > > > Or, is there any chance you all could persuade the python-daemon > > maintainer to update his install_requires so that it doesn't include > > setuptools? > > Maybe? In this case given cutting a dependency is tractable I'd be > okay absorbing one under my maintenance for now. > > Do you have any interest in submitting a patch that does as you say > about the daemon bits? > > (You might wonder why that's even there: it's for the > apparently-very-useful pipelined and parallel WAL download on restore) > -- You received this message because you are subscribed to the Google Groups "wal-e" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
