RE:building manpages via setup.py
Ok, If I replace
'{interpreter} setup.py build_man'
by
'env; setup.py build_man'
I get this
HOME=/home/picca
PYTHONPATH=/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build
but If I read the code of
def create_pydistutils_cfg(func):
"""distutils doesn't have sane command-line API - this decorator creates
.pydistutils.cfg file to workaround it
hint: if you think this is plain stupid, please don't read
distutils/setuptools/distribute sources
"""
def wrapped_func(self, context, args, *oargs, **kwargs):
fpath = join(args['home_dir'], '.pydistutils.cfg')
if not exists(fpath):
with open(fpath, 'w', encoding='utf-8') as fp:
lines = ['[clean]\n',
'all=1\n',
'[build]\n',
'build-lib={}\n'.format(args['build_dir']),
'[install]\n',
'force=1\n',
'install-layout=deb\n',
'install-scripts=/usr/bin\n',
'install-lib={}\n'.format(args['install_dir']),
'[easy_install]\n',
'allow_hosts=None\n']
log.debug('pydistutils config file:\n%s', ''.join(lines))
fp.writelines(lines)
context['ENV']['HOME'] = args['home_dir']
return func(self, context, args, *oargs, **kwargs)
we can see that the HOME should be home_dir
HOME should be /home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7
I think that there is a problem here.
right ?
RE:building manpages via setup.py
So I am sltill investigating
with some debug output, I can see this the first build part gives
dh_auto_build -- --after-build '{interpreter} setup.py build_man'
pybuild --build -i python{version} -p 2.7 --after-build "{interpreter}
setup.py build_man"
D: pybuild pybuild:474: version: 2.20170125
D: pybuild pybuild:475: ['/usr/bin/pybuild', '--build', '-i',
'python{version}', '-p', '2.7', '--after-build', '{interpreter} setup.py
build_man']
D: pybuild pybuild:36: cfg: Namespace(after_build='{interpreter} setup.py
build_man', after_clean=None, after_configure=None, after_install=None,
after_test=None, before_build=None, before_clean=None, before_configure=None,
before_install=None, before_test=None, build_args=None, build_only=True,
clean_args=None, clean_only=False, configure_args=None, configure_only=False,
custom_tests=False, destdir='debian/tmp', detect_only=False,
dir='/home/picca/Debian/silx/silx', disable=None, ext_destdir=None,
ext_pattern='\\.so(\\.[^/]*)?$', install_args=None, install_dir=None,
install_only=False, interpreter=['python{version}'], list_systems=False,
name='silx', quiet=False, really_quiet=False, system=None, test_args=None,
test_nose=False, test_only=False, test_pytest=False, test_tox=False,
verbose=True, versions=['2.7'])
D: pybuild pybuild:103: detected build system: distutils (certainty: 61%)
I: pybuild base:184: /usr/bin/python setup.py build
D: pybuild tools:217: invoking: /usr/bin/python setup.py build
INFO:silx.setup:Use setuptools
running build
running build_py
creating /home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build/silx
<- right path
the after part execute the build_man part BUT
I: pybuild pybuild:242: python2.7 setup.py build_man
D: pybuild tools:217: invoking: python2.7 setup.py build_man
INFO:silx.setup:Use setuptools
running build_man
running build_py
creating build/lib.linux-i386-2.7
creating build/lib.linux-i386-2.7/silx <- ko
So the after part does not seems to take into account the .pydistutils.cfg file
???
Is it normal ?
RE:building manpages via setup.py
Hello Piotr,
I am struggling, with the build system.
I will speak about this solution
> | override_dh_auto_build:
> | dh_auto_build -- --after-build '{interpreter} setup.py build_man'
the code instrumented of the BuildMan is this one
class BuildMan(Command):
"""Command to build man pages"""
user_options = []
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
build = self.get_finalized_command('build')
path = sys.path
path.insert(0, os.path.abspath(build.build_lib))
print("PYTHONPATH", os.environ.get("PYTHONPATH", None))
print("sys.path", sys.path, os.path.exists(build.build_lib))
env = dict((str(k), str(v)) for k, v in os.environ.items())
env["PYTHONPATH"] = os.pathsep.join(path)
print("env[PYTHONPATH]", env["PYTHONPATH"])
import subprocess
status = subprocess.call(["mkdir", "-p", "build/man"])
if status != 0:
raise RuntimeError("Fail to create build/man directory")
try:
import tempfile
import stat
script_name = None
# help2man expect a single executable file to extract the help
# we create it, execute it, and delete it at the end
# create a launcher using the right python interpreter
script_fid, script_name = tempfile.mkstemp(prefix="%s_" % PROJECT,
text=True)
script = os.fdopen(script_fid, 'wt')
script.write("#!%s\n" % sys.executable)
script.write("import runpy\n")
script.write("runpy.run_module('%s', run_name='__main__')\n" %
PROJECT)
script.close()
# make it executable
mode = os.stat(script_name).st_mode
os.chmod(script_name, mode + stat.S_IEXEC)
# execute help2man
p = subprocess.Popen(["help2man", script_name, "-o",
"build/man/silx.1"], env=env)
status = p.wait()
if status != 0:
raise RuntimeError("Fail to generate man documentation")
finally:
# clean up the script
if script_name is not None:
os.remove(script_name)
When I run the build it generate a pydistconfig file wit this content
D: pybuild plugin_distutils:55: pydistutils config file:
[clean]
all=1
[build]
build-lib=/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build
[install]
force=1
install-layout=deb
install-scripts=/usr/bin
install-lib=/usr/lib/python2.7/dist-packages
[easy_install]
allow_hosts=None
But when it comes to the --after step, I get this output
('PYTHONPATH', '/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build')
('sys.path', ['/home/picca/Debian/silx/silx/build/lib.linux-i386-2.7',
'/home/picca/Debian/silx/silx',
'/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build',
'/usr/lib/python2.7', '/usr/lib/python2.7/plat-i386-linux-gnu',
'/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old',
'/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages/PILcompat',
'/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7'],
False)
('env[PYTHONPATH]',
'/home/picca/Debian/silx/silx/build/lib.linux-i386-2.7:/home/picca/Debian/silx/silx:/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build:/usr/lib/python2.7:/usr/lib/python2.7/plat-i386-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload:/usr/local/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages/PILcompat:/usr/lib/python2.7/dist-packages/gtk-2.0:/usr/lib/pymodules/python2.7')
So as you can see the path returned by build.build_lib
seems to be
/home/picca/Debian/silx/silx/build/lib.linux-i386-2.7'
and not
/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build (provided via pydist)
So my question is what is wrong.
self.get_finalized_command('build')
doesn not take into account the pydistconfig file ?
Thanks for your help
Frederic
RE:building manpages via setup.py
> The snippet you quoted is not specific to extension modules but to the > use of the autodoc feature, which requires the modules to be in the > PYTHONPATH. The `sys.path.insert` hack is just here so that you don't > have to specify PYTHONPATH yourself when running the upstream Makefile. I just says that this snipset is in the conf.py and I yes we need this for autodoc. > I don't understand how setup.py / build options are affecting the Sphinx > documentation. You are supposed to either call the generator manually > via sphinx-build (which the style guide recommends), or using the > upstream Makefile (which upstream often does). autodoc introspect the module so you need a working modules from python, whcih is avhieved via the right PYTHONPATH. So, during the build phase you can gives options on the python setup.py command line. This has an effect on the location where the module and the extensions are built. build seems to be the default location BUT when we use pybuild it is under .pybuild So the default PYTHONPATH does not work. there is no way for sphinx to know where the module and its extensions are excepty by providing the right PYTHONPATH. Except if sphinx would provide a build_sphinx command whcih take care of this problem and take into account the parameters of the setup.py. > As far as reliability is concerned, my reference here is upstream. If > upstream can produce the docs, then I should be able to do it too. And > they don't have pybuild for that. Yes but sometime there si some hack done by the upstream in conf.py or setup.py in order to look for the module at the standards path (under build/) which doesn not work with pybuild. > Why would you build the docs for each supported Python version, > considering you'll end up shipping only one instance of the generated > HTML? I am probably missing something here. I use this as a sort of unit test. And it is interesting because I catche a bunch of python2/python3 issues like this. Then I can find issues between sphinx running on python2 or python3. This is at least a good way to check thaht all the modules can be laoded. autodoc is interesting for this. > You might as well generate the docs just once with the default Python 2 > or Python 3 interpreter, the same way you would do without extension > modules, no? > Pardon my insistence, but I really fail to understand what issues you > are referring to. incompatibilities vetween python2 and python 3 code. peoples trie to write code python2 and python3 compatible but this is not thaht easy... > More than I expected, indeed. > And amongst these 643 packages, how many have such large build times > that the overhead of an additional inplace call would be considered > prohibitive, I wonder? I do not know, I just think that we should be able to automatically generate packages for python modules from pip in order to reduce the effort of pacakging, So there should be only one way to build the documentation , manpages. Escpecillay wheren there is extensions. > Besides, the docs will typically be processed by arch-all builders > (provided you use -indep targets), so arch-any builds won't even feel > that overhead, right? Yes except that this is ot because --inplace work that the result of python setup.py build are identical. it is possible to forget a module referenced by another module in the setup.py. It workd with --inplace but it does not work for real after build ans installation. Cheers Fred I could be wrong though.
RE:building manpages via setup.py
> First, that's very speculative. Second, that's upstream's problem.
> The upstream Makefile and conf.py are likely generated by Sphinx itself
> via sphinx-quickstart. Did your upstream tinker with them that much that
> they cannot be trusted?
No this is just that does not fit well with extension.
> Some upstream do use a build_sphinx command, but it is far from common
> and it does not solve the extension module problem.
Yes this is what they are doing. extension are part of the python ec-system but
it seems thaht this is not well suported out of the box by sphinx.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
You need to customize the sys.path in order to find the extensions.
This path can change depending on the setup.py build options so this is not
reliable.
> Why? All you need is *one* occurrence of the extension modules somewhere
> in the PYTHONPATH in order to generate the docs. Chances are that's how
> upstream generates them.
Because by experience I find issue in the build system and the python code when
building the doc for multiple
interpreter (python2/python3 differences...)
> > Found a total of 643 reverse build-depend(s) for python-all-dev.
> >
> > not that small
> How is the ratio over all the Python packages? I suspect very small.
Found a total of 1968 reverse build-depend(s) for python-all.
not that small 32 % ;)
Cheers
Fred
RE:building manpages via setup.py
> PYTHONPATH=. sphinx-build -N -b html > One can also use the sphinx-generated Makefile if available: > PYTHONPATH=$(CURDIR) $(MAKE) -C html > Both are simple one-liners and do not rely on pybuild. Yes it works but this is fragile since the organisation of the module can change in the sources. at least the .pybuild directory is under the responsability of pybuild and we should use pybuild instead of relying on the maintainer snipset. (typo error, change during the time.) It would be nice if the doc generation in python shold be standardize. > If it does not cost much to build the extension inplace, then the > simplest option is to prefix one of these calls above with: > python3 setup.py build_ext --inplace when you have multiple verison of the interpreter you prefer to avoid --inplace. > If the cost is prohibitive, which arguably applies to a very limited set > of packages (yours included) then you would use pybuild for that, as > Piotr kindly suggested. Yes it depends olsa of the arch. Some are really slow > Considering the rarity of this use case though, I wonder whether it is > worth adding a separate section to the style guide. Found a total of 643 reverse build-depend(s) for python-all-dev. not that small cheers Fred
RE:building manpages via setup.py
> At the end of the day, it is just a matter of providing an appropriate > PYTHONPATH, regardless of whether pybuild is used or not. Yes but to avoid the multiplications of way to provide this PYTHONPATH. Is it possible to have the recommended way which works for modules and extensions. once agreed, we should put this in the wiki Cheers Fred
RE:building manpages via setup.py
> Perhaps the LibraryStyleGuide should be updated to reflect on this > change? I believe we are still advising explicit http_proxy / > https_proxy exports prior to running sphinx-build. And running sphinx-build does not work expecially if there is extensions in the documentation. sphinx-build should be run via pybuild in order to know about the build_dir. right ? Fred
RE:building manpages via setup.py
> export PYBUILD_AFTER_BUILD={interpreter} setup.py build_man
Thanks, The only drawback I see with this solution is that I want to run
dh_auto_build 2 times,
- 1 for the arch part (pyhon modules, extensions and manpages)
- 2 for the indep part (doc build)
nevertheless thans a lot it is flexxible enought for my case.
> you can drop it, PYTHONPATH and http_proxy should be set by pybuild
Is it true for jessie
I need to support jessie and stretch
And even debian7...
Fred
RE:building manpages via setup.py
Hello piotr
> if you want to test all requested Python interpreters:
I prefer this solution in order to check that the built extensions are working
for all the python interpreters.
The doc use autodoc so it is nice to have this functionnality
| override_dh_auto_build:
| dh_auto_build -- --after-build '{interpreter} setup.py build_man'
It is possible to replace this by
export PYBUILD_AFTER_BUILD='{interpreter} setup.py build_man'
> there's no need to prepend --after-build argument with
> "env PYTHONPATH={build_dir}; " - pybuild sets that to build dir automatically
Do you mean that I should remove the PYTHONPATH in this ?
override_dh_sphinxdoc
ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
PYBUILD_SYSTEM=custom \
PYBUILD_BUILD_ARGS="cd doc && PYTHONPATH={build_dir}
http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source build/html"
dh_auto_build # HTML generator
dh_installdocs "doc/build/html" -p python-pyfai-doc
dh_sphinxdoc -O--buildsystem=pybuild
endif
--
GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
RE:building manpages via setup.py
> Simplest I can think of would be to build the extensions inplace
> followed by the call to build_man. Something like:
> override_dh_auto_build:
> dh_auto_build
> python3 setup.py build_ext --inplace
> python3 setup.py build_man
I do not want to build once more the ext (it takes a lot's of time on my
computer and it is a wast of ressources).
I just want to run {interpreter} setup.py build_man
where
are provided by pybuild
> I left the http_proxy exports and nodoc guards out for clarity.
thanks ;)
> Second questions what is the right way to generate the man pages for a
> python application ?
> Usually via Sphinx if the upstream documentation uses it. Regardless of
> the stack, `help2man` is often considered the poor man choice for
> generating manpages.
Thanks, I know this but I need more time to convince the upstream to find a
better solution :)
> Let me know if you'd like me to have a look :-)
thanks a lot
Fred
