[Distutils] Easy Install / Setup Tools Example

[Todd Greenwood-Geer]

I've taken to heart your suggestions. The result is what I hope is a very simple example / tutorial that demonstrates:

 * the file structure of a project destined to be an egg
 * how to use doctests and unit tests with setuptools / eggs
 * how to build and deploy an egg
 * how to register with pypi
 * how to (and not to) generate wrapper scripts
 * how to write documentation using restructured text

Instructions:
$ easy_install SimpleExampleEgg

That's it. You have now installed v 0.2. The tutorial docs
are the egg:

/usr/lib/python2.4/site-packages/SimpleExampleEgg-0.2-py2.4.egg:fruit/docs/readme.html

Could I get a quick review of this? I'm attaching the readme.html.

-Todd

Phillip J. Eby wrote:

At 08:27 PM 12/8/2005 -0800, Todd Greenwood-Geer wrote:
small_unittest.py::

        import small
        import unittest
        import doctest

        def getTestSuite():
                suite = unittest.TestSuite()
                for mod in small,:
                        suite.addTest(doctest.DocTestSuite(mod))
                return suite

        runner = unittest.TextTestRunner()
        runner.run(getTestSuite())

If 'small.small_unittest' is the actual name of the module, then doing this:

python -c "from unittest import main; main(None)" small.small_unittest.getTestSuite will run the test, even if it's inside an egg. No unzipping is necessary. If the egg wasn't installed with a .pth file (i.e. --multi-version or to a non-standard location), you'll need to make sure it's on PYTHONPATH, along with its dependencies.

Here is what the src tree for the simple example looks like:

dir listing::

        small/src/small/test/test.txt
        small/src/small/small2.py
        small/src/small/small.py
        small/src/small/small_unittest.pyc
        small/src/small/__init__.py
        small/src/small/small_unittest.py
        small/src/small_unittest.py

This looks like a duplicate. I'm also not sure why you have a small2, or why you bother having a separate test subdirectory. Indeed, for this "small" of an example, a src/ directory is probably overkill. For clarity, I'd also suggest calling the topmost directory something like SmallExample to help distinguish your project name (SmallExample) from your package name (small).

        small/setup_mult.py

What's this file for?

        small/ez_setup.py

http://peak.telecommunity.com/DevCenter/setuptools#using-setuptools-without-bundling-it

If you're using Subversion, btw, you can use the svn:externals trick so that you don't have to manually maintain this file; see the setuptools manual under:

The thing to note here is how small.small and small.small2 are referenced::

        from small import small as s1
        from small import small2 as s2

For some reason, this gave me plenty of problems.

It's generally not a good idea to name a module the same as a package, and definitely not the same as a class within the module, or else it gets unclear in code which one you're talking about. In a language like Java, there's no such ambiguity because syntactically you can't refer to a class where a package should be or vice versa, but in Python there are only objects, so you should not name them all the same thing or you (and/or the interpreter) will be confused. :) Similarly, I suspect that your example has way too many files perhaps because you have a Java background? It really isn't necessary to split Python projects up into so many small pieces; it just makes more work for you and doesn't get you any benefit until the files get too large to conveniently work with.

Complex Case
------------

The complex case splits out the test modules into a nested package hierarchy like this:

...

Package Hierarchy
+++++++++++++++++

package hierarchy::

        small/src/
        small/src/size/
        small/src/size/small
        small/src/size/large

Ouch. I thought the previous example was the complex one. ;) Seriously, your simple example is way more complex than it needs to be. This bigger one makes my head hurt, so I'm not going to try to comment on it further, except to suggest that its top-level directory should be named "small-mult-test" since that's your project name. Also, I'm guessing you mean your package hiearchy is size, size.small, and size.large. The small/src stuff isn't part of the package hierarchy, just the directory hierarchy. By the way, the only reason to have a 'src' directory under your main project directory is if you have a bunch of stuff that's *not* source code there. Such as documentation, documentation directories, test directories, examples, etc. Your projects here have none of those things, only a setup script, so there's no reason to have a 'src' subdirectory; you could just put the 'small' or 'size' package directly under the project directory, and in fact this is the most common layout for published Python packages. The 'src' or 'lib' subdirectory approach is mainly used by large projects with a lot of other things in their project directory, and by people who got used to that format by working on large projects. :)

        from size.small import small
        from size.large import large

I said I wasn't going to comment further, but this is technically a repetition of my earlier comment: please don't name packages and modules the same thing, you will confuse yourself and everyone else who will never be sure if 'small' means 'size.small' or 'size.small.small' or worse, 'size.small.small.small'. Eeek!

* url : need to see if that works for downloading dependencies (next tutorial, not this one)

Yes, it does, *if* you register your package with PyPI. The URL given will be scanned for download links when and if easy_install looks for your package on PyPI.

Title: SimpleExampleEgg

SimpleExampleEgg

Author:	Todd Greenwood
Title:	SimpleExampleEgg
Keywords:	python, egg, setuptools, distutils, tutorial, doctests, unit tests, deploy, install, example
Version:	0.1
License:	GPL v. 2
Date:	Dec 2005

This aims to be a super simple example of python, setuptools, docutils, unit-tests, and eggs.

Contents

• SimpleExampleEgg
• Introduction
• Basic Source Files
• Tests and Building
• Building the REST docs
• Backtracking
• The ez_setup.py file
• Create a PiPy account:
• App Deployment
• Script Generation
• Building
• Deploy Locally (Dev Mode)
• Deploy To pipy
• End User App Install
• User Install
• User Run Unit Tests
• Conclusion

Introduction

Hopefully, this will clarify, in my own mind, at least, how to make super simple apps that have all sorts of cool extras such as working unit tests, launch scripts, deploy and install flawlessly...etc. Ideally, I'll be able to write apps that 'just work'. So my end users can just point some tool (easy_install) at a file (an egg) and voila, a working app.

Here's what I'd like to have happen for this exercise:

• user clicks a link

• app downloads and installs

• user runs the app's internal test suite

Along the way, I'll play around with:

• deploying the app locally, and to pipy

• user may recieve the app via email, a web link, file copy, etc.

• doctests and unit tests

Note, this documentation was written using Mikolaj Machowski's Vim reStructured Text plugin (vst). It is the coolest thing since the 'intraweb'. Look at the source 'readme.rest' and the generated 'readme.html'. The code listings and sample code output were generated as part of the build, very little 'copy and paste' here.

Basic Source Files

1. dir listing:

   apple.py
   docs
   __init__.py
   orange.py
   simpletests.py
   

2. apple.py:

   import sys
   
   def doConsole(): print make_pie('CONSOLE')
   
   def make_pie(who):
   	"""
   >>> import apple as a
   >>> a.make_pie('Todd')
   'Todd likes pie!!!'
   	"""
   	return '%s likes pie!!!'%who
   #
   def _test():
   	import doctest
   	return doctest.testmod()
   #
   if __name__ == "__main__":
       _test()
       if len(sys.argv)>1: print make_pie(sys.argv[1])
       
   
   

3. Let's run this tiny app to see what we get:

   $ python apple.py Mr.Guido

   Mr.Guido likes pie!!!
   

4. setup.py:

   from ez_setup import use_setuptools
   use_setuptools()
   from setuptools import setup, find_packages
   #
   setup(name = "SimpleExampleEgg",
       version = "0.1",
       description = "test",
       author = "Todd Greenwood",
       author_email = "[EMAIL PROTECTED]",
   	entry_points = {'console_scripts': [
   		'make_apple_pie = fruit.apple:doConsole'
   		]},
       packages = find_packages(exclude=['ez_setup'] ),
   	package_data = {'':['docs/*.html', 'docs/*.rest','docs/*.vim']},
       test_suite = 'fruit.simpletests.getTestSuite',
       license = "GNU Lesser General Public License",
       classifiers = [
           "Development Status :: 3 - Alpha",
           "Intended Audience :: Developers",
           "License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)",
           "Programming Language :: Python",
           "Topic :: Utilities",
       ],
       zip_safe=True,
       )
       
   
   

5. simpletests.py:

   import apple
   import unittest
   import doctest
   #
   def getTestSuite():
   	suite = unittest.TestSuite()
   	for mod in apple,:
   	    suite.addTest(doctest.DocTestSuite(mod))
   	return suite
   #
   runner = unittest.TextTestRunner()
   runner.run(getTestSuite())
       
   
   

Tests and Building

1. Let's run the test suite:

   $ python simpletests.py

   .
   ----------------------------------------------------------------------
   Ran 1 test in 0.007s
   
   OK
   

2. Now, let's run the test suite from setuptools:

   $ python setup.py test

   running test
   running egg_info
   writing ./SimpleExampleEgg.egg-info/PKG-INFO
   writing top-level names to ./SimpleExampleEgg.egg-info/top_level.txt
   writing entry points to ./SimpleExampleEgg.egg-info/entry_points.txt
   running build_ext
   .
   ----------------------------------------------------------------------
   Ran 1 test in 0.008s
   
   OK
   Doctest: fruit.apple.make_pie ... ok
   
   ----------------------------------------------------------------------
   Ran 1 test in 0.001s
   
   OK
   

3. Next, let's make sure the build system works. But first, I should mention that this documentation is generated via a vim pluggin.

Building the REST docs

1. So, while from a command line you would execute:

   $ python setup bdist_egg

   Because this doc lives in ./fruit/docs/readme.rest, I will use the pbu utility to walk the directory tree and launch setup.py. If you would like to build the docs, here is the process:

   1. install pbu:

       $ easy_install buildutils
      

   2. build the egg using the 'pbu' utility:

       $ cd ./fruit/docs
       $ pbu dbist_egg
      

2. So that's exactly what the docs do. So, to see all this in action, build the docs :

    $ vim readme.rest -s runvim.vim
   
    BTW - runvim.vim is just a quickie script to execute :Vsti within vim,
    write, save, and exit vim.
   

3. Back to the build system, let's do a build :

   running bdist_egg
   running egg_info
   writing ./SimpleExampleEgg.egg-info/PKG-INFO
   writing top-level names to ./SimpleExampleEgg.egg-info/top_level.txt
   writing entry points to ./SimpleExampleEgg.egg-info/entry_points.txt
   installing library code to build/bdist.linux-i686/egg
   running install_lib
   warning: install_lib: 'build/lib' does not exist -- no Python modules to install
   creating build/bdist.linux-i686/egg
   creating build/bdist.linux-i686/egg/EGG-INFO
   copying ./SimpleExampleEgg.egg-info/PKG-INFO -> build/bdist.linux-i686/egg/EGG-INFO
   copying ./SimpleExampleEgg.egg-info/top_level.txt -> build/bdist.linux-i686/egg/EGG-INFO
   copying ./SimpleExampleEgg.egg-info/entry_points.txt -> build/bdist.linux-i686/egg/EGG-INFO
   creating 'dist/SimpleExampleEgg-0.1-py2.4.egg' and adding 'build/bdist.linux-i686/egg' to it
   removing 'build/bdist.linux-i686/egg' (and everything under it)
   

4. And lastly, let's check to make sure that we have what we expect in the output:

   ``unzip -l /home/tgreenwo/active/SimpleExampleEgg/dist/SimpleExampleEgg-0.1-py2.4.egg'``

   Archive:  /home/tgreenwo/active/SimpleExampleEgg/dist/SimpleExampleEgg-0.1-py2.4.egg
     Length     Date   Time    Name
    --------    ----   ----    ----
         290  12-14-05 12:11   fruit/orange.py
         330  12-14-05 15:33   fruit/apple.py
         240  12-14-05 12:26   fruit/simpletests.py
           0  12-14-05 12:13   fruit/__init__.py
         556  12-14-05 16:24   fruit/orange.pyc
         754  12-14-05 16:24   fruit/apple.pyc
         537  12-14-05 16:24   fruit/simpletests.pyc
         124  12-14-05 16:24   fruit/__init__.pyc
       18896  12-14-05 14:54   fruit/docs/readme.html
        8768  12-14-05 16:02   fruit/docs/readme.rest
         141  12-14-05 12:36   fruit/docs/vstrc.vim
          14  12-14-05 14:32   fruit/docs/runvim.vim
         489  12-14-05 16:24   EGG-INFO/PKG-INFO
           6  12-14-05 16:24   EGG-INFO/top_level.txt
          58  12-14-05 16:24   EGG-INFO/entry_points.txt
           0  12-14-05 16:24   EGG-INFO/zip-safe
    --------                   -------
       31203                   16 files
   

Backtracking

Some things I should mention....

• apply the svn 'trick' to get the ez_setup file

• get a freebie ftp account to test uploading to

• create a pipy account so that you can upload there, too

The ez_setup.py file

1. Per the Peak website, install ez_setup.py as an svn external thingy. Here are the steps:

    $ cd SimpleExampleEgg/
    $ svn propedit svn:externals .
    # enter this in the editor: 
    ez_setup svn://svn.eby-sarna.com/svnroot/ez_setup
    # slurp down the latest ez_setup file  
    $ svn update
    # you should see the ez_setup dir now..., if you don't, try
    $ svn propget svn:externals
    ez_setup svn://svn.eby-sarna.com/svnroot/ez_setup
   

2. It would be nice to be able to create a project that just housed the ez_setup project, and then just reference that project in all my other projects. However, apparently svn does not support chained external dependencies. So each project will either have to have a copy of ez_setup.py, or this external reference.

Create a PiPy account:

1. this part is easy:

    $ python setup.py register
   

2. now just follow the instructions..:

    running register
    We need to know who you are, so please choose either:
    1. use your existing login,
    2. register as a new user,
    3. have the server generate a new password for you (and email it to you),
      or
    4. quit
    Your selection [default 1]:
   

3. you'll need your username and password the first time you deploy. subsequent deploys will store your log in info somewhere in your home dir

App Deployment

Now, we want to do your basic 'app' stuff:

• script generation

• build

• deploy

• test

Script Generation

The idea here is that if I have some cool command line app, then how do I access it when it's all tucked inside an egg? The generated wrapper scripts do just that.

1. add entry points to setup.py:

    setup(
    ...
    entry_points = {'console_scripts': [
        'make_apple_pie = fruit.apple:make_pie'
        ]},
   

2. note that you cannot pass command line vars to the method:

    tgreenwo@luxor~/active/SimpleExampleEgg$ make_apple_pie
    Traceback (most recent call last):
     File "/usr/bin/make_apple_pie", line 7, in ?
         sys.exit(
         TypeError: make_pie() takes exactly 1 argument (0 given)
         tgreenwo@luxor~/active/SimpleExampleEgg$ python make_apple_pie
         python: can't open file 'make_apple_pie': [Errno 2] No such file or
         directory
    tgreenwo@luxor~/active/SimpleExampleEgg$ which make_apple_pie
         /usr/bin/make_apple_pie
    tgreenwo@luxor~/active/SimpleExampleEgg$ cat /usr/bin/make_apple_pie
         #!/home/tgreenwo/bin/python
         # EASY-INSTALL-ENTRY-SCRIPT:
         # 'SimpleExampleEgg==0.1','console_scripts','make_apple_pie'
         __requires__ = 'SimpleExampleEgg==0.1'
         import sys
         from pkg_resources import load_entry_point
   
         sys.exit(
            load_entry_point('SimpleExampleEgg==0.1', 'console_scripts',
            'make_apple_pie')()
            )
   

3. add a method to apple.py w/o params:

    #apple.py
    def doConsole(): print make_pie('CONSOLE')
   
    #setup.py
    entry_points = {'console_scripts': [
        'make_apple_pie = fruit.apple:doConsole'
        ]},
   

4. install (as root) again

   easy_install dist/SimpleExampleEgg-0.1-py2.4.egg

5. and run again:

    tgreenwo@luxor~/active/SimpleExampleEgg$ /usr/bin/make_apple_pie
    CONSOLE likes pie!!!
   

Building

We did this already, but, we'll do it again here.

1. Build the app into an egg:

   $ python setup bdist_egg

   running bdist_egg
   running egg_info
   writing ./SimpleExampleEgg.egg-info/PKG-INFO
   writing top-level names to ./SimpleExampleEgg.egg-info/top_level.txt
   writing entry points to ./SimpleExampleEgg.egg-info/entry_points.txt
   installing library code to build/bdist.linux-i686/egg
   running install_lib
   warning: install_lib: 'build/lib' does not exist -- no Python modules to install
   creating build/bdist.linux-i686/egg
   creating build/bdist.linux-i686/egg/EGG-INFO
   copying ./SimpleExampleEgg.egg-info/PKG-INFO -> build/bdist.linux-i686/egg/EGG-INFO
   copying ./SimpleExampleEgg.egg-info/top_level.txt -> build/bdist.linux-i686/egg/EGG-INFO
   copying ./SimpleExampleEgg.egg-info/entry_points.txt -> build/bdist.linux-i686/egg/EGG-INFO
   creating 'dist/SimpleExampleEgg-0.1-py2.4.egg' and adding 'build/bdist.linux-i686/egg' to it
   removing 'build/bdist.linux-i686/egg' (and everything under it)
   

2. And lastly, let's check to make sure that we have what we expect:

   $ unzip -l /home/tgreenwo/active/SimpleExampleEgg/dist/SimpleExampleEgg-0.1-py2.4.egg

   Archive:  /home/tgreenwo/active/SimpleExampleEgg/dist/SimpleExampleEgg-0.1-py2.4.egg
     Length     Date   Time    Name
    --------    ----   ----    ----
         290  12-14-05 12:11   fruit/orange.py
         330  12-14-05 15:33   fruit/apple.py
         240  12-14-05 12:26   fruit/simpletests.py
           0  12-14-05 12:13   fruit/__init__.py
         556  12-14-05 16:24   fruit/orange.pyc
         754  12-14-05 16:24   fruit/apple.pyc
         537  12-14-05 16:24   fruit/simpletests.pyc
         124  12-14-05 16:24   fruit/__init__.pyc
       18896  12-14-05 14:54   fruit/docs/readme.html
        8768  12-14-05 16:02   fruit/docs/readme.rest
         141  12-14-05 12:36   fruit/docs/vstrc.vim
          14  12-14-05 14:32   fruit/docs/runvim.vim
         489  12-14-05 16:24   EGG-INFO/PKG-INFO
           6  12-14-05 16:24   EGG-INFO/top_level.txt
          58  12-14-05 16:24   EGG-INFO/entry_points.txt
           0  12-14-05 16:24   EGG-INFO/zip-safe
    --------                   -------
       31203                   16 files
   

Deploy Locally (Dev Mode)

1. create a deployment directory to house the egg links:

    $ cd working
    $ mkdir deploy
   

2. add the 'deploy' directory to your pythonpath:

   		export PYTHONPATH=/home/tgreenwo/working/twisted:/home/tgreenwo/working/django:/home/tgreenwo/working/deploy
   

3. install the egg locally in dev mode:

   $ python setup.py develop

   running develop
   running egg_info
   writing ./SimpleExampleEgg.egg-info/PKG-INFO
   writing top-level names to ./SimpleExampleEgg.egg-info/top_level.txt
   writing entry points to ./SimpleExampleEgg.egg-info/entry_points.txt
   running build_ext
   Creating /usr/lib/python2.4/site-packages/SimpleExampleEgg.egg-link (link to .)
   error: /usr/lib/python2.4/site-packages/SimpleExampleEgg.egg-link: Permission denied
   

4. this time, we specify the deploy directory:

   $ python setup.py develop -d /home/tgreenwo/working/deploy

   running develop
   running egg_info
   writing ./SimpleExampleEgg.egg-info/PKG-INFO
   writing top-level names to ./SimpleExampleEgg.egg-info/top_level.txt
   writing entry points to ./SimpleExampleEgg.egg-info/entry_points.txt
   running build_ext
   Creating /home/tgreenwo/data/working/deploy/SimpleExampleEgg.egg-link (link to .)
   Installing make_apple_pie script to /home/tgreenwo/working/deploy
   
   Installed /home/tgreenwo/active/SimpleExampleEgg/fruit/docs
   
   Because this distribution was installed --multi-version or --install-dir,
   before you can import modules from this package in an application, you
   will need to 'import pkg_resources' and then use a 'require()' call
   similar to one of these examples, in order to select the desired version:
   
       pkg_resources.require("SimpleExampleEgg")  # latest installed version
       pkg_resources.require("SimpleExampleEgg==0.1")  # this exact version
       pkg_resources.require("SimpleExampleEgg>=0.1")  # this version or higher
   
   Processing dependencies for SimpleExampleEgg==0.1
   

Deploy To pipy

1. build and deploy in one step:

    $ setup.py register sdist bdist_egg upload
   

2. I had problems, so I enabled the following:

    $ python setup.py register sdist bdist_egg upload --show-response
   

3. Once you noodle thru all the errors for your log in, password, and whether all the meta information like url and topic are set, then the egg will upload.

End User App Install

General steps are:

• su or sudo

• easy_install [egg name] or

• easy_install [pipy project name]

• run unit tests

User Install

1. Simple. One step:

    root$ easy_install SimpleExampleEgg
   
    Searching for SimpleExampleEgg
    Reading http://www.python.org/pypi/SimpleExampleEgg/
    Best match: SimpleExampleEgg 0.1
    Downloading
    http://cheeseshop.python.org/packages/2.4/S/SimpleExampleEgg/SimpleExampleEgg-0.1-py2.4.egg#md5=d53e38ea496e03d30b632dc08265128b
    Processing SimpleExampleEgg-0.1-py2.4.egg
    Moving SimpleExampleEgg-0.1-py2.4.egg to /usr/lib/python2.4/site-packages
    Adding SimpleExampleEgg 0.1 to easy-install.pth file
    Installing make_apple_pie script to /usr/bin
   
    Installed /usr/lib/python2.4/site-packages/SimpleExampleEgg-0.1-py2.4.egg
    Processing dependencies for SimpleExampleEgg
   

2. done

User Run Unit Tests

1. this should run the unit tests for the installed app:

    $ python -c "from unittest import main; main(None)"
    fruit.simpletests.getTestSuite
   

2. let's try that:

   python -c 'from unittest import main; main(None)' fruit.simpletests.getTestSuite

   .
   ----------------------------------------------------------------------
   Ran 1 test in 0.007s
   
   OK
   .
   ----------------------------------------------------------------------
   Ran 1 test in 0.001s
   
   OK
   

Conclusion

1. vim + vst rocks

2. eggs are cool

signature.asc
Description: OpenPGP digital signature

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig