Re: dh-python (pybuild + dh_py*) documentation

2015-10-28 Thread Matthias Klose

On 26.10.2015 12:36, Piotr Ożarowski wrote:

there are manpages: pybuild¹, dh_python3² / dh_python2³ / dh_pypy⁴
there's a wiki page⁵ with examples, there's even
/usr/share/doc/dh-python/README.PyDist⁶ and a talk⁷ about pybuild during
DebConf... but people still tell me dh-python's documentation sucks
so...


talks and mailing list discussions are forgotten.


How can I improve it? Do you need more detailed description of options?
Do you need more examples? Or maybe I should hide most of options, the
"corner case" ones? I focus on making things work out of the box, if
possible, and make it very customizable so that weird corner cases are
possible as well - this means there are a lot of options, is this the
problem?

Can you be more specific about what's missing?


Agreed on the high level overview.  Maybe somebody could come up with a table of 
contents, and then people can start filling that? There was at least one comment 
from a book author ;)


I'd like to see a section with examples too, maybe even referencing particular 
packages, which show good practice for a given use case, or which demonstrate 
some feature.


Please don't hide things.  When I see something in a build log, I'd like to be 
able to reproduce it.  pybuild writes these "I: ..." messages, but these are 
still incomplete. It would be good to see directory changes and other side 
effects like setting of env vars and written config files as well.  I found 
these missing while looking at #803242. However asking you on irc usually helps 
;)  Having these build logs a bit more verbose won't hurt.


The #803242 is exposed by having packages built with multiple python versions, 
so maybe it would have been found by a testsuite, or some autopkg tests.  A 
testsuite for the pybuild features would be very nice to have.


Matthias



Re: dh-python (pybuild + dh_py*) documentation

2015-10-28 Thread Piotr Ożarowski
> > Can you be more specific about what's missing?
> 
> I for one lack a high level presentation of how the various bits work together

here's what I got so far (I wanted to provide it as
/usr/share/doc/dh-python/README). I stopped working on it because now I
have to prove dh-python is not responsible for maintainers that override
dh_auto_install and rm egg-info or new "features" in python3.5
-- 
evil general Piotr (Brutus: or is it caezar now?)
dh-python
=

dh-python provides various tools that help packaging Python related files in 
Debian.

* pybuild is a tool that implements dh sequencer's dh_auto_foo commands (but it
  can be used outside dh as well). It builds and installs files.

* dh_python2 / dh_python3 / dh_pypy are tools that take what pybuild produces
  and generate runtime dependencies, maintainer scripts (and fix some common
  mistakes, like installing files into site-packages instead of dist-packages,
  /usr/local/bin/ shebangs, removes .py files from -dbg packages, etc.)

  To translate requires.txt (file installed in dist-packages/foo.egg-info/)
  into Debian dependencies, a list of packages that provide given Egg
  distribution is used. If dependency is not found there, dpkg -S is used (i.e.
  given dependency has to be installed, you need it in Build-Depends in order
  to run tests anyway). See `dependencies` section in dh_python3's manpage for
  more details.

  * dh_python2 works on ./debian/python-foo/ files and other binary packages 
that
have ${python:Depends} in Depends fields.
It ignores Python 3.X and PyPy specific directories.
See dh_python2 manpage for more details.
  
  * dh_python3 works on ./debian/python3-foo/ files and other binary packages 
that
have ${python3:Depends} in Depends fields.
It ignores Python 2.X and PyPy specific directories.
See dh_python3 manpage for more details.
  
  * dh_pypy works on ./debian/pypy-foo/ files and other binary packages that
have ${pypy:Depends} in Depends fields.
It ignores Python 2.X and Python 3.X specific directories.
See dh_pypy manpage for more details.


how it works together
--

Simplified workflow looks like this:

.. code:: python

# dh_auto_clean stage
for interpreter in REQUESTED_INTERPRETERS:
for version in REQUESTED_VERSIONS:
PYBUILD_BEFORE_CLEAN
pybuild --clean
PYBUILD_AFTER_CLEAN

plenty_of_other_dh_foo_tools_invoked_here

# dh_auto_configure stage
for interpreter in REQUESTED_INTERPRETERS:
for version in REQUESTED_VERSIONS:
PYBUILD_BEFORE_CONFIGURE
pybuild --configure
PYBUILD_AFTER_CONFIGURE

plenty_of_other_dh_foo_tools_invoked_here

# dh_auto_build stage
for interpreter in REQUESTED_INTERPRETERS:
for version in REQUESTED_VERSIONS:
PYBUILD_BEFORE_BUILD
pybuild --build
PYBUILD_AFTER_BUILD

plenty_of_other_dh_foo_tools_invoked_here

# dh_auto_test stage
for interpreter in REQUESTED_INTERPRETERS:
for version in REQUESTED_VERSIONS:
PYBUILD_BEFORE_TEST
pybuild --test
PYBUILD_AFTER_TEST

plenty_of_other_dh_foo_tools_invoked_here

# dh_auto_install stage
for interpreter in REQUESTED_INTERPRETERS:
for version in REQUESTED_VERSIONS:
PYBUILD_BEFORE_INSTALL
pybuild --install
PYBUILD_AFTER_INSTALL

plenty_of_other_dh_foo_tools_invoked_here

dh_python2
dh_python3
dh_pypy

plenty_of_other_dh_foo_tools_invoked_here


pybuild --$step
~~~

This command is autodetected, it currently supports distutils, autotools, cmake
and a custom build system where you can define your own set of commands. Why do
we need it? dh_auto_foo doesn't know each command has to be invoked for each
interpreter and version.


REQUESTED_INTERPRETERS
~~

is parsed from Build-Depends if --buildsystem=pybuild is set, if it's not: you
have to pass --interpreter to pybuild (more in its manpage)

  * python{3,}-all{,-dev} - all cPython interpreters (for packages that provide
public modules / extensions)
  * python{3,}-all-dbg - all cPython debug interpreters (if -dbg package is 
provided)
  * python{3,} - default cPython or closest to default interpreter only (use
this if you build Python application)
  * python{3,}-dbg - default cPython debug (or closest to the default one) only
  * pypy - PyPy interpreter (use this )


REQUESTED_VERSIONS
~~

is parsed from X-Python{,3}-Version and Build-Depends (the right X-*-Version is
parsed based on interpreters listed in B-D, see above) See also Debian Python
Policy for X-Python-Version description.


BEFORE and AFTER commands
~

can be different for each interpreter and/or version examples:

 * `PYBUILD_AFTER_BUILD_python3.5=rm {destdir}/{build_dir}/foo/bar2X.py`
 * 

Re: dh-python (pybuild + dh_py*) documentation

2015-10-28 Thread Barry Warsaw
On Oct 28, 2015, at 05:59 PM, Piotr Ożarowski wrote:

>Each day I choose a victim and enable Piotr's evil mode.

Beware the Eye of Piotr.

:)

-Barry



Re: dh-python (pybuild + dh_py*) documentation

2015-10-27 Thread Barry Warsaw
On Oct 26, 2015, at 06:28 PM, Raphael Hertzog wrote:

>I for one lack a high level presentation of how the various bits work together
>and a clear description of the "magic" behind each tool.

I agree with this.  When you know where to look, the documentation is actually
quite good.  Maybe there are a few things here or there that could be filled
in or improved, but it's all generally there.

The problem is that when things don't build as expected, it's difficult to
understand which magical piece is misbehaving.

Cheers,
-Barry



Re: dh-python (pybuild + dh_py*) documentation

2015-10-26 Thread Raphael Hertzog
Hi Piotr,

On Mon, 26 Oct 2015, Piotr Ożarowski wrote:
> How can I improve it? Do you need more detailed description of options?
> Do you need more examples? Or maybe I should hide most of options, the
> "corner case" ones? I focus on making things work out of the box, if
> possible, and make it very customizable so that weird corner cases are
> possible as well - this means there are a lot of options, is this the
> problem?
> 
> Can you be more specific about what's missing?

I for one lack a high level presentation of how the various bits work together
and a clear description of the "magic" behind each tool.

For instance, dh_python2 explains that it tries to translate Python
dependencies into Debian dependencies but it's not clear that it needs
the corresponding packages installed (so it's not clear that for this
to work you have to add something to Build-Depends).

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/