[Distutils] Best practices on distributing documentation and examples with eggs?
On Thu, July 26, 2007 9:50 pm, Phillip J. Eby wrote: At 02:13 PM 7/26/2007 -0500, Dave Peterson wrote: In further discussions at Enthought, we're now thinking that creating a 'docs' dir as a top-level under python itself makes sense, and then the installation of eggs could copy docs and/or examples there in a manner similar to how it handles scripts into the platforms scripts dir. It would make sense that they should be put in dirs named with a format like project name-version so that the user could differentiate the docs for each project they installed. If this was done, it seems like it wouldn't be too hard for Stan, and other rpm builders, to either create symbolic links or copy these to /usr/share/doc in order to maintain compatibility with the LSB. This isn't a bad idea, although I think that there are some hurdles that need to be worked out. My general thought is that, were easy_install to support documentation, it would need to take an option to determine the documentation base directory, and have a --no-docs install option. One issue is that under most RPM systems, package names are more like 'python-foobar-1.2', and 'foobar' may not even be the exact distutils name of the package. So, installing the docs to the correct location in the general case may be more complex. Note, however, that bdist_rpm offers a --doc-files option, which of course can be configured via setup.cfg. I have no idea how it works, though, so don't ask me. :) The practice of changing the name of the whatever package to python-whatever may be distro dependent. Also (thankfully), python-whatever is installed in site-packages/whatever and not in site-packages/python-whatever. It was not always that way. Back at something like RedHat 7, they did change the names and the installation locations. If you had another program that imported the package, you sometimes had to go in and change the code to match their changed name. I can understand the reasons of the distros and repositories for doing this (keeping all the python or other language packages together in the list) but that does not make it less annoying. In looking to see if a package is in the repository, you have to look in various places beyond what the original developer/maintainer of the package calls it. Also, I'm not sure of the split between /usr/share/whatever and usr/share/doc/whatever for things like docs, tests, examples, and other miscellaneous files. It tends to be at the whim of the packager. In terms of what bdist_rpm does, it seems to be in the preparation of the rpm spec file. The spec file contains scripts (some of which point to macros in /usr/lib/rpm/macros on a Fedora system) and bdist_rpm seems to build the scripts and execute them using rpm-build. The build and install scripts do python setup.py build or python setup.py install with whatever options are specified. I think the actual location of the files may be governed by the %files section of the spec, but I will have to spend some time looking at other spec files to be sure that is where it happens. Stan Klein On Thu, July 26, 2007 12:46 pm, Phillip J. Eby wrote: At 02:13 PM 7/26/2007 -0500, Dave Peterson wrote: In further discussions at Enthought, we're now thinking that creating a 'docs' dir as a top-level under python itself makes sense, and then the installation of eggs could copy docs and/or examples there in a manner similar to how it handles scripts into the platforms scripts dir. It would make sense that they should be put in dirs named with a format like project name-version so that the user could differentiate the docs for each project they installed. If this was done, it seems like it wouldn't be too hard for Stan, and other rpm builders, to either create symbolic links or copy these to /usr/share/doc in order to maintain compatibility with the LSB. This isn't a bad idea, although I think that there are some hurdles that need to be worked out. My general thought is that, were easy_install to support documentation, it would need to take an option to determine the documentation base directory, and have a --no-docs install option. One issue is that under most RPM systems, package names are more like 'python-foobar-1.2', and 'foobar' may not even be the exact distutils name of the package. So, installing the docs to the correct location in the general case may be more complex. Note, however, that bdist_rpm offers a --doc-files option, which of course can be configured via setup.cfg. I have no idea how it works, though, so don't ask me. :) The practice of changing the name of the whatever package to python-whatever may be distro dependent. Also (thankfully), python-whatever is installed in site-packages/whatever and not in site-packages/python-whatever. It was not always that way. Back at something like RedHat 7, they did change the names and the installation locations. If you had something else that imported the package, you had to go in
Re: [Distutils] Best practices on distributing documentation and examples with eggs?
Phillip J. Eby wrote: At 02:55 PM 7/26/2007 -0400, Barry Warsaw wrote: On Jul 26, 2007, at 11:52 AM, Phillip J. Eby wrote: There are a few different ways you could do this. The easiest would be to put a docs subdirectory either inside one of your packages or inside your .egg-info directory, then use the pkg_resources resource or metadata APIs to list and extract them. One advantage to using something like .egg-info/docs would be that this could perhaps be recognized by some sort of standard tools in the future. I always thought .egg-info was just an artifact of the build. No - it's also a place for metadata that doesn't live under a specific package directory. Recommending that things get put there means it should get version controlled. Yep. But isn't .egg-info completely recreatable through setup.py? Given that all it really does is present information from setup.py in a different manner, I've always been putting it on my svn:ignore list and only version-controlled the canonical source -- setup.py. That's becaue I can see and have seen the information get out of synch easily, e.g. if somebody changes setup.py but forgets to run setup.py egg_info. -- http://worldcookery.com -- Professional Zope documentation and training ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Best practices on distributing documentation and examples with eggs?
At 08:48 AM 7/31/2007 +0200, Philipp von Weitershausen wrote: But isn't .egg-info completely recreatable through setup.py? Not if you're putting other data there, no. Chandler, for example, has an i18n framework that stores message catalogs and other localized resources there. Of course, it's certainly possible to use egg_info.writers plugins that would copy those things there, but it's a lot easier for development if you don't, as it would require a build step every time you changed something. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Best practices on distributing documentation and examples with eggs?
Over on the enthought-dev mailing list we're having a bit of a discussion on what the best way to distribute documenation and examples is for projects that we distribute binary eggs for. The general consensus is that it would be very nice indeed if there was a way to generate a tarball, or platform install, of just documentation and examples so that people wouldn't need to download a full, presumably significantly larger, source tarball. Another option would be that eggs included the documentation and examples and that, during the installation of the egg, those docs and examples were relocated to a common location (outside of the zip) to make access by users more convenient. This latter idea is similar to how Ruby Gems deal with docs. I don't claim to be a distutils or setuptools guru, so it shouldn't be too surprising that I can't seem to find anything about a setuptools or distutils command to do either of these. Am I missing something? If not, does it seem like something that might be worthy of putting into setuptools? -- Dave ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Best practices on distributing documentation and examples with eggs?
At 02:13 PM 7/26/2007 -0500, Dave Peterson wrote: In further discussions at Enthought, we're now thinking that creating a 'docs' dir as a top-level under python itself makes sense, and then the installation of eggs could copy docs and/or examples there in a manner similar to how it handles scripts into the platforms scripts dir. It would make sense that they should be put in dirs named with a format like project name-version so that the user could differentiate the docs for each project they installed. If this was done, it seems like it wouldn't be too hard for Stan, and other rpm builders, to either create symbolic links or copy these to /usr/share/doc in order to maintain compatibility with the LSB. This isn't a bad idea, although I think that there are some hurdles that need to be worked out. My general thought is that, were easy_install to support documentation, it would need to take an option to determine the documentation base directory, and have a --no-docs install option. One issue is that under most RPM systems, package names are more like 'python-foobar-1.2', and 'foobar' may not even be the exact distutils name of the package. So, installing the docs to the correct location in the general case may be more complex. Note, however, that bdist_rpm offers a --doc-files option, which of course can be configured via setup.cfg. I have no idea how it works, though, so don't ask me. :) One other thought we've been throwing around is an idea of adding something to easy-install so that users can still get a minimal binary distribution if they want, and then, if they want docs, run a command like: easy_install --get-docs enthought.traits where easy_install would then introspect the enthought.traits egg and figure out how to retrieve the docs and install them into ../python/docs. Actually, if the default is for easy_install to install documentation, then simply rerunning easy_install should suffice (easy_install on an already-installed package makes the package active and installs scripts, so installing docs should work the same way.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Best practices on distributing documentation and examples with eggs?
At 04:07 PM 7/26/2007 -0400, Barry Warsaw wrote: On Jul 26, 2007, at 3:44 PM, Phillip J. Eby wrote: Note that if you are using a revision control system, and setuptools deletes a file from there, it might get restored by your revision control, and that could potentially be a problem unless you also manually delete the file. When does setuptools delete a file there? Whenever a setuptools egg_info.writers plugin does. :) More usefully, most files generated by setup() arguments (e.g. entry_points) will be deleted the next time you run egg_info, if you removed that argument from setup(). ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Best practices on distributing documentation and examples with eggs?
On Thu, July 26, 2007 12:46 pm, Phillip J. Eby wrote: At 03:02 AM 7/26/2007 -0500, Dave Peterson wrote: Over on the enthought-dev mailing list we're having a bit of a discussion on what the best way to distribute documenation and examples is for projects that we distribute binary eggs for. The general consensus is that it would be very nice indeed if there was a way to generate a tarball, or platform install, of just documentation and examples so that people wouldn't need to download a full, presumably significantly larger, source tarball. Another option would be that eggs included the documentation and examples and that, during the installation of the egg, those docs and examples were relocated to a common location (outside of the zip) to make access by users more convenient. This latter idea is similar to how Ruby Gems deal with docs. I don't claim to be a distutils or setuptools guru, so it shouldn't be too surprising that I can't seem to find anything about a setuptools or distutils command to do either of these. Am I missing something? There are a few different ways you could do this. The easiest would be to put a docs subdirectory either inside one of your packages or inside your .egg-info directory, then use the pkg_resources resource or metadata APIs to list and extract them. One advantage to using something like .egg-info/docs would be that this could perhaps be recognized by some sort of standard tools in the future. If not, does it seem like something that might be worthy of putting into setuptools? It might be worth establishing convention(s) for, yes. Tools could then be developed around them, including perhaps changes to easy_install. This relates to a question I asked this list earlier this month (but didn't get a response). For Linux systems the Linux Standards Base references the Unix Filesystem Hierarchy Standard (that applies to all Unix systems as well). The FHS specifies that documentation files (other than specially formatted items like man pages) go into /usr/share/doc/package_name_and_version These sometimes include examples, demos, and similar files. For example, the docs on my FC5 system for inkscape go in /usr/share/doc/inkscape-0.45.1. In that case the doc files are a typical minimal set: /usr/share/doc/inkscape-0.45.1/AUTHORS /usr/share/doc/inkscape-0.45.1/COPYING /usr/share/doc/inkscape-0.45.1/ChangeLog /usr/share/doc/inkscape-0.45.1/NEWS /usr/share/doc/inkscape-0.45.1/README and the examples, tutorials, clipart, and many miscellaneous files are in /usr/share/inkscape. The actual executables are in /usr/bin. In some cases the documentation is created as a separate package. For example, Python does this for its html-based docs and on my FC5 system, the python html docs are in /usr/share/doc/python-docs-2.4.3/. Similar considerations go to configuration files that are supposed to go into /etc. There are a number of other rules, and they are generally observed by systems that use rpm and deb packaging such as Fedora and Ubuntu. I couldn't figure out how to make this happen when using bdist_rpm, which is why I asked the earlier question. It seems to me that the only way using current Python packaging features would be to put the docs somewhere in the Python site-packages hierarchy and do a post-install scripted move to where they belong under the LSB/FHS rules. It would be preferable to get them to go where they belong without the need for post-install scripting. Stan Klein ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Best practices on distributing documentation and examples with eggs?
Stanley A. Klein wrote: On Thu, July 26, 2007 12:46 pm, Phillip J. Eby wrote: At 03:02 AM 7/26/2007 -0500, Dave Peterson wrote: Over on the enthought-dev mailing list we're having a bit of a discussion on what the best way to distribute documenation and examples is for projects that we distribute binary eggs for. The general consensus is that it would be very nice indeed if there was a way to generate a tarball, or platform install, of just documentation and examples so that people wouldn't need to download a full, presumably significantly larger, source tarball. Another option would be that eggs included the documentation and examples and that, during the installation of the egg, those docs and examples were relocated to a common location (outside of the zip) to make access by users more convenient. This latter idea is similar to how Ruby Gems deal with docs. I don't claim to be a distutils or setuptools guru, so it shouldn't be too surprising that I can't seem to find anything about a setuptools or distutils command to do either of these. Am I missing something? There are a few different ways you could do this. The easiest would be to put a docs subdirectory either inside one of your packages or inside your .egg-info directory, then use the pkg_resources resource or metadata APIs to list and extract them. One advantage to using something like .egg-info/docs would be that this could perhaps be recognized by some sort of standard tools in the future. If not, does it seem like something that might be worthy of putting into setuptools? It might be worth establishing convention(s) for, yes. Tools could then be developed around them, including perhaps changes to easy_install. This relates to a question I asked this list earlier this month (but didn't get a response). For Linux systems the Linux Standards Base references the Unix Filesystem Hierarchy Standard (that applies to all Unix systems as well). The FHS specifies that documentation files (other than specially formatted items like man pages) go into /usr/share/doc/package_name_and_version These sometimes include examples, demos, and similar files. For example, the docs on my FC5 system for inkscape go in /usr/share/doc/inkscape-0.45.1. In that case the doc files are a typical minimal set: /usr/share/doc/inkscape-0.45.1/AUTHORS /usr/share/doc/inkscape-0.45.1/COPYING /usr/share/doc/inkscape-0.45.1/ChangeLog /usr/share/doc/inkscape-0.45.1/NEWS /usr/share/doc/inkscape-0.45.1/README and the examples, tutorials, clipart, and many miscellaneous files are in /usr/share/inkscape. The actual executables are in /usr/bin. In some cases the documentation is created as a separate package. For example, Python does this for its html-based docs and on my FC5 system, the python html docs are in /usr/share/doc/python-docs-2.4.3/. Similar considerations go to configuration files that are supposed to go into /etc. There are a number of other rules, and they are generally observed by systems that use rpm and deb packaging such as Fedora and Ubuntu. I couldn't figure out how to make this happen when using bdist_rpm, which is why I asked the earlier question. It seems to me that the only way using current Python packaging features would be to put the docs somewhere in the Python site-packages hierarchy and do a post-install scripted move to where they belong under the LSB/FHS rules. It would be preferable to get them to go where they belong without the need for post-install scripting. In further discussions at Enthought, we're now thinking that creating a 'docs' dir as a top-level under python itself makes sense, and then the installation of eggs could copy docs and/or examples there in a manner similar to how it handles scripts into the platforms scripts dir. It would make sense that they should be put in dirs named with a format like project name-version so that the user could differentiate the docs for each project they installed. If this was done, it seems like it wouldn't be too hard for Stan, and other rpm builders, to either create symbolic links or copy these to /usr/share/doc in order to maintain compatibility with the LSB. One other thought we've been throwing around is an idea of adding something to easy-install so that users can still get a minimal binary distribution if they want, and then, if they want docs, run a command like: easy_install --get-docs enthought.traits where easy_install would then introspect the enthought.traits egg and figure out how to retrieve the docs and install them into ../python/docs. If we found time to work on implementing something, is there a preference for which path we should pursue? -- Dave ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Best practices on distributing documentation and examples with eggs?
At 02:55 PM 7/26/2007 -0400, Barry Warsaw wrote: On Jul 26, 2007, at 11:52 AM, Phillip J. Eby wrote: There are a few different ways you could do this. The easiest would be to put a docs subdirectory either inside one of your packages or inside your .egg-info directory, then use the pkg_resources resource or metadata APIs to list and extract them. One advantage to using something like .egg-info/docs would be that this could perhaps be recognized by some sort of standard tools in the future. I always thought .egg-info was just an artifact of the build. No - it's also a place for metadata that doesn't live under a specific package directory. Recommending that things get put there means it should get version controlled. Yep. But I've found that occasionally I have to blow away .egg-info files to get 'setup.py develop' to work properly (i.e. like a 'make clean' step). Really? I'd consider that to be a bug. Could you perhaps be more specific? Note that if you are using a revision control system, and setuptools deletes a file from there, it might get restored by your revision control, and that could potentially be a problem unless you also manually delete the file. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
