Re: dh-python (pybuild + dh_py*) documentation

2015-10-28 Thread Matthias Klose

On 26.10.2015 12:36, Piotr Ożarowski wrote:

there are manpages: pybuild¹, dh_python3² / dh_python2³ / dh_pypy⁴
there's a wiki page⁵ with examples, there's even
/usr/share/doc/dh-python/README.PyDist⁶ and a talk⁷ about pybuild during
DebConf... but people still tell me dh-python's documentation sucks
so...


talks and mailing list discussions are forgotten.


How can I improve it? Do you need more detailed description of options?
Do you need more examples? Or maybe I should hide most of options, the
"corner case" ones? I focus on making things work out of the box, if
possible, and make it very customizable so that weird corner cases are
possible as well - this means there are a lot of options, is this the
problem?

Can you be more specific about what's missing?


Agreed on the high level overview.  Maybe somebody could come up with a table of 
contents, and then people can start filling that? There was at least one comment 
from a book author ;)


I'd like to see a section with examples too, maybe even referencing particular 
packages, which show good practice for a given use case, or which demonstrate 
some feature.


Please don't hide things.  When I see something in a build log, I'd like to be 
able to reproduce it.  pybuild writes these "I: ..." messages, but these are 
still incomplete. It would be good to see directory changes and other side 
effects like setting of env vars and written config files as well.  I found 
these missing while looking at #803242. However asking you on irc usually helps 
;)  Having these build logs a bit more verbose won't hurt.


The #803242 is exposed by having packages built with multiple python versions, 
so maybe it would have been found by a testsuite, or some autopkg tests.  A 
testsuite for the pybuild features would be very nice to have.


Matthias



Re: dh-python (pybuild + dh_py*) documentation

2015-10-28 Thread Piotr Ożarowski
> > Can you be more specific about what's missing?
> 
> I for one lack a high level presentation of how the various bits work together

here's what I got so far (I wanted to provide it as
/usr/share/doc/dh-python/README). I stopped working on it because now I
have to prove dh-python is not responsible for maintainers that override
dh_auto_install and rm egg-info or new "features" in python3.5
-- 
evil general Piotr (Brutus: or is it caezar now?)
dh-python
=

dh-python provides various tools that help packaging Python related files in 
Debian.

* pybuild is a tool that implements dh sequencer's dh_auto_foo commands (but it
  can be used outside dh as well). It builds and installs files.

* dh_python2 / dh_python3 / dh_pypy are tools that take what pybuild produces
  and generate runtime dependencies, maintainer scripts (and fix some common
  mistakes, like installing files into site-packages instead of dist-packages,
  /usr/local/bin/ shebangs, removes .py files from -dbg packages, etc.)

  To translate requires.txt (file installed in dist-packages/foo.egg-info/)
  into Debian dependencies, a list of packages that provide given Egg
  distribution is used. If dependency is not found there, dpkg -S is used (i.e.
  given dependency has to be installed, you need it in Build-Depends in order
  to run tests anyway). See `dependencies` section in dh_python3's manpage for
  more details.

  * dh_python2 works on ./debian/python-foo/ files and other binary packages 
that
have ${python:Depends} in Depends fields.
It ignores Python 3.X and PyPy specific directories.
See dh_python2 manpage for more details.
  
  * dh_python3 works on ./debian/python3-foo/ files and other binary packages 
that
have ${python3:Depends} in Depends fields.
It ignores Python 2.X and PyPy specific directories.
See dh_python3 manpage for more details.
  
  * dh_pypy works on ./debian/pypy-foo/ files and other binary packages that
have ${pypy:Depends} in Depends fields.
It ignores Python 2.X and Python 3.X specific directories.
See dh_pypy manpage for more details.


how it works together
--

Simplified workflow looks like this:

.. code:: python

# dh_auto_clean stage
for interpreter in REQUESTED_INTERPRETERS:
for version in REQUESTED_VERSIONS:
PYBUILD_BEFORE_CLEAN
pybuild --clean
PYBUILD_AFTER_CLEAN

plenty_of_other_dh_foo_tools_invoked_here

# dh_auto_configure stage
for interpreter in REQUESTED_INTERPRETERS:
for version in REQUESTED_VERSIONS:
PYBUILD_BEFORE_CONFIGURE
pybuild --configure
PYBUILD_AFTER_CONFIGURE

plenty_of_other_dh_foo_tools_invoked_here

# dh_auto_build stage
for interpreter in REQUESTED_INTERPRETERS:
for version in REQUESTED_VERSIONS:
PYBUILD_BEFORE_BUILD
pybuild --build
PYBUILD_AFTER_BUILD

plenty_of_other_dh_foo_tools_invoked_here

# dh_auto_test stage
for interpreter in REQUESTED_INTERPRETERS:
for version in REQUESTED_VERSIONS:
PYBUILD_BEFORE_TEST
pybuild --test
PYBUILD_AFTER_TEST

plenty_of_other_dh_foo_tools_invoked_here

# dh_auto_install stage
for interpreter in REQUESTED_INTERPRETERS:
for version in REQUESTED_VERSIONS:
PYBUILD_BEFORE_INSTALL
pybuild --install
PYBUILD_AFTER_INSTALL

plenty_of_other_dh_foo_tools_invoked_here

dh_python2
dh_python3
dh_pypy

plenty_of_other_dh_foo_tools_invoked_here


pybuild --$step
~~~

This command is autodetected, it currently supports distutils, autotools, cmake
and a custom build system where you can define your own set of commands. Why do
we need it? dh_auto_foo doesn't know each command has to be invoked for each
interpreter and version.


REQUESTED_INTERPRETERS
~~

is parsed from Build-Depends if --buildsystem=pybuild is set, if it's not: you
have to pass --interpreter to pybuild (more in its manpage)

  * python{3,}-all{,-dev} - all cPython interpreters (for packages that provide
public modules / extensions)
  * python{3,}-all-dbg - all cPython debug interpreters (if -dbg package is 
provided)
  * python{3,} - default cPython or closest to default interpreter only (use
this if you build Python application)
  * python{3,}-dbg - default cPython debug (or closest to the default one) only
  * pypy - PyPy interpreter (use this )


REQUESTED_VERSIONS
~~

is parsed from X-Python{,3}-Version and Build-Depends (the right X-*-Version is
parsed based on interpreters listed in B-D, see above) See also Debian Python
Policy for X-Python-Version description.


BEFORE and AFTER commands
~

can be different for each interpreter and/or version examples:

 * `PYBUILD_AFTER_BUILD_python3.5=rm {destdir}/{build_dir}/foo/bar2X.py`
 * 

Re: dh-python (pybuild + dh_py*) documentation

2015-10-28 Thread Barry Warsaw
On Oct 28, 2015, at 05:59 PM, Piotr Ożarowski wrote:

>Each day I choose a victim and enable Piotr's evil mode.

Beware the Eye of Piotr.

:)

-Barry