Re: building manpages via setup.py
On 02/08/17 10:45, PICCA Frederic-Emmanuel wrote:
First, that's very speculative. Second, that's upstream's problem.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
You need to customize the sys.path in order to find the extensions.
The snippet you quoted is not specific to extension modules but to the
use of the autodoc feature, which requires the modules to be in the
PYTHONPATH. The `sys.path.insert` hack is just here so that you don't
have to specify PYTHONPATH yourself when running the upstream Makefile.
This path can change depending on the setup.py build options so this is not
reliable.
I don't understand how setup.py / build options are affecting the Sphinx
documentation. You are supposed to either call the generator manually
via sphinx-build (which the style guide recommends), or using the
upstream Makefile (which upstream often does).
As far as reliability is concerned, my reference here is upstream. If
upstream can produce the docs, then I should be able to do it too. And
they don't have pybuild for that.
Why? All you need is *one* occurrence of the extension modules somewhere
in the PYTHONPATH in order to generate the docs. Chances are that's how
upstream generates them.
Because by experience I find issue in the build system and the python code when
building the doc for multiple
interpreter (python2/python3 differences...)
Why would you build the docs for each supported Python version,
considering you'll end up shipping only one instance of the generated
HTML? I am probably missing something here.
You might as well generate the docs just once with the default Python 2
or Python 3 interpreter, the same way you would do without extension
modules, no?
Pardon my insistence, but I really fail to understand what issues you
are referring to.
Found a total of 643 reverse build-depend(s) for python-all-dev.
not that small
How is the ratio over all the Python packages? I suspect very small.
Found a total of 1968 reverse build-depend(s) for python-all.
not that small 32 % ;)
More than I expected, indeed.
And amongst these 643 packages, how many have such large build times
that the overhead of an additional inplace call would be considered
prohibitive, I wonder?
Besides, the docs will typically be processed by arch-all builders
(provided you use -indep targets), so arch-any builds won't even feel
that overhead, right?
I could be wrong though.
Re: building manpages via setup.py
On 02/08/17 09:55, PICCA Frederic-Emmanuel wrote: PYTHONPATH=. sphinx-build -N -b html One can also use the sphinx-generated Makefile if available: PYTHONPATH=$(CURDIR) $(MAKE) -C html Both are simple one-liners and do not rely on pybuild. Yes it works but this is fragile since the organisation of the module can change in the sources. First, that's very speculative. Second, that's upstream's problem. I have not seen many upstream projects playing with the layout of their modules from one version to the next. If that's the case, then there are worst things to be worried about (API breakage for instance) than the docs. at least the .pybuild directory is under the responsability of pybuild and we should use pybuild instead of relying on the maintainer snipset. (typo error, change during the time.) The upstream Makefile and conf.py are likely generated by Sphinx itself via sphinx-quickstart. Did your upstream tinker with them that much that they cannot be trusted? It would be nice if the doc generation in python shold be standardize. Some upstream do use a build_sphinx command, but it is far from common and it does not solve the extension module problem. If it does not cost much to build the extension inplace, then the simplest option is to prefix one of these calls above with: python3 setup.py build_ext --inplace when you have multiple verison of the interpreter you prefer to avoid --inplace. Why? All you need is *one* occurrence of the extension modules somewhere in the PYTHONPATH in order to generate the docs. Chances are that's how upstream generates them. I fail to picture how this is an issue in practice considering your build override will run the isolated pybuilds first and the sphinx call last. If the cost is prohibitive, which arguably applies to a very limited set of packages (yours included) then you would use pybuild for that, as Piotr kindly suggested. Yes it depends olsa of the arch. Some are really slow That's unfortunate, indeed. Considering the rarity of this use case though, I wonder whether it is worth adding a separate section to the style guide. Found a total of 643 reverse build-depend(s) for python-all-dev. not that small How is the ratio over all the Python packages? I suspect very small.
Re: building manpages via setup.py
On 02/08/17 09:19, PICCA Frederic-Emmanuel wrote: At the end of the day, it is just a matter of providing an appropriate PYTHONPATH, regardless of whether pybuild is used or not. Yes but to avoid the multiplications of way to provide this PYTHONPATH. For the vast majority of packages, the current method listed in LibraryStyleGuide applies, i.e.: PYTHONPATH=. sphinx-build -N -b html One can also use the sphinx-generated Makefile if available: PYTHONPATH=$(CURDIR) $(MAKE) -C html Both are simple one-liners and do not rely on pybuild. Is it possible to have the recommended way which works for modules and extensions. If it does not cost much to build the extension inplace, then the simplest option is to prefix one of these calls above with: python3 setup.py build_ext --inplace If the cost is prohibitive, which arguably applies to a very limited set of packages (yours included) then you would use pybuild for that, as Piotr kindly suggested. once agreed, we should put this in the wiki Considering the rarity of this use case though, I wonder whether it is worth adding a separate section to the style guide. Ghis
Re: building manpages via setup.py
On 02/08/17 09:03, PICCA Frederic-Emmanuel wrote: Perhaps the LibraryStyleGuide should be updated to reflect on this change? I believe we are still advising explicit http_proxy / https_proxy exports prior to running sphinx-build. And running sphinx-build does not work expecially if there is extensions in the documentation. sphinx-build should be run via pybuild in order to know about the build_dir. right ? At the end of the day, it is just a matter of providing an appropriate PYTHONPATH, regardless of whether pybuild is used or not. The default is $(CURDIR) for the vast majority of packages. But, it may also be something else if extension packages are involved, or if the package directory is under a different folder than the root directory, such as src/. Ghis
Re: building manpages via setup.py
On 02/08/17 08:44, Piotr Ożarowski wrote: [PICCA Frederic-Emmanuel, 2017-08-02] you can drop it, PYTHONPATH and http_proxy should be set by pybuild Is it true for jessie I need to support jessie and stretch And even debian7... I didn't test it even for unstable, but IIRC pybuild exports those in all steps since a long time ago. You'll know after first builds... Perhaps the LibraryStyleGuide should be updated to reflect on this change? I believe we are still advising explicit http_proxy / https_proxy exports prior to running sphinx-build. Ghis
Re: building manpages via setup.py
[PICCA Frederic-Emmanuel, 2017-08-02] > > you can drop it, PYTHONPATH and http_proxy should be set by pybuild > > Is it true for jessie > > I need to support jessie and stretch > > And even debian7... I didn't test it even for unstable, but IIRC pybuild exports those in all steps since a long time ago. You'll know after first builds... PS please don't CC me, I'm subscribing the list -- GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: building manpages via setup.py
[PICCA Frederic-Emmanuel, 2017-08-02]
> > if you want to test all requested Python interpreters:
>
> I prefer this solution in order to check that the built extensions are
> working for all the python interpreters.
> The doc use autodoc so it is nice to have this functionnality
>
> | override_dh_auto_build:
> | dh_auto_build -- --after-build '{interpreter} setup.py build_man'
>
>
> It is possible to replace this by
>
> export PYBUILD_AFTER_BUILD='{interpreter} setup.py build_man'
export PYBUILD_AFTER_BUILD={interpreter} setup.py build_man
(i.e. without quotes if you want to use Make's export)
> > there's no need to prepend --after-build argument with
> > "env PYTHONPATH={build_dir}; " - pybuild sets that to build dir
> > automatically
>
> Do you mean that I should remove the PYTHONPATH in this ?
you can drop it, PYTHONPATH and http_proxy should be set by pybuild
> override_dh_sphinxdoc
> ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
> PYBUILD_SYSTEM=custom \
> PYBUILD_BUILD_ARGS="cd doc && PYTHONPATH={build_dir}
> http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source build/html"
> dh_auto_build # HTML generator
> dh_installdocs "doc/build/html" -p python-pyfai-doc
> dh_sphinxdoc -O--buildsystem=pybuild
> endif
--
GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: building manpages via setup.py
[PICCA Frederic-Emmanuel, 2017-08-01]
> This pacakge contain one module with extensions (the important point)
>
> The new upstream version 0.14.0 provide a build_man target via the setup.py
>
> So in ordert to generate the doc I need to do
>
> python setup.py build_man
>
[...]
> So what should I do to run
>
> python setup.py build_man with the options provided by pybuild during the
> normal build in order to let the generated script find the pyFAI modules and
> its extensions ?
if you want it to be run only once (for default Python 3 interpreter):
| override_dh_auto_build:
| dh_auto_build
| pybuild --system=custom --build --build-args '{interpreter} setup.py
build_man' --interpreter=python3
if you want to test all requested Python interpreters:
| override_dh_auto_build:
| dh_auto_build -- --after-build '{interpreter} setup.py build_man'
there's no need to prepend --after-build argument with
"env PYTHONPATH={build_dir}; " - pybuild sets that to build dir automatically
--
GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: building manpages via setup.py
On 01/08/17 15:15, PICCA Frederic-Emmanuel wrote:
Hello,
I am working on the pyfai package.
This pacakge contain one module with extensions (the important point)
The new upstream version 0.14.0 provide a build_man target via the setup.py
So in ordert to generate the doc I need to do
python setup.py build_man
Now if I look at this target, I can find this code
-
class BuildMan(Command):
"""Command to build man pages"""
user_options = []
def initialize_options(self):
pass
def finalize_options(self):
pass
def entry_points_iterator(self):
"""Iterate other entry points available on the project."""
entry_points = self.distribution.entry_points
console_scripts = entry_points.get('console_scripts', [])
gui_scripts = entry_points.get('gui_scripts', [])
scripts = []
scripts.extend(console_scripts)
scripts.extend(gui_scripts)
for script in scripts:
elements = script.split("=")
target_name = elements[0].strip()
elements = elements[1].split(":")
module_name = elements[0].strip()
function_name = elements[1].strip()
yield target_name, module_name, function_name
def run(self):
build = self.get_finalized_command('build')
path = sys.path
path.insert(0, os.path.abspath(build.build_lib))
env = dict((str(k), str(v)) for k, v in os.environ.items())
env["PYTHONPATH"] = os.pathsep.join(path)
import subprocess
status = subprocess.call(["mkdir", "-p", "build/man"])
if status != 0:
raise RuntimeError("Fail to create build/man directory")
import tempfile
import stat
script_name = None
entry_points = self.entry_points_iterator()
for target_name, module_name, function_name in entry_points:
logger.info("Build man for entry-point target '%s'" % target_name)
# help2man expect a single executable file to extract the help
# we create it, execute it, and delete it at the end
py3 = sys.version_info >= (3, 0)
try:
# create a launcher using the right python interpreter
script_fid, script_name = tempfile.mkstemp(prefix="%s_" %
target_name, text=True)
script = os.fdopen(script_fid, 'wt')
script.write("#!%s\n" % sys.executable)
script.write("import %s as app\n" % module_name)
script.write("app.%s()\n" % function_name)
script.close()
# make it executable
mode = os.stat(script_name).st_mode
os.chmod(script_name, mode + stat.S_IEXEC)
# execute help2man
man_file = "build/man/%s.1" % target_name
command_line = ["help2man", script_name, "-o", man_file]
if not py3:
# Before Python 3.4, ArgParser --version was using
# stderr to print the version
command_line.append("--no-discard-stderr")
p = subprocess.Popen(command_line, env=env)
status = p.wait()
if status != 0:
raise RuntimeError("Fail to generate '%s' man
documentation" % target_name)
finally:
# clean up the script
if script_name is not None:
os.remove(script_name)
-
As you can see this create a launch script for each entry point found in the
setup.py and run help2man on it.
For now I would like to use this setup.py without modification.
So what should I do to run
python setup.py build_man with the options provided by pybuild during the
normal build in order to let the generated script find the pyFAI modules and
its extensions ?
Simplest I can think of would be to build the extensions inplace
followed by the call to build_man. Something like:
override_dh_auto_build:
dh_auto_build
python3 setup.py build_ext --inplace
python3 setup.py build_man
I left the http_proxy exports and nodoc guards out for clarity.
Second questions what is the right way to generate the man pages for a python
application ?
Usually via Sphinx if the upstream documentation uses it. Regardless of
the stack, `help2man` is often considered the poor man choice for
generating manpages.
Let me know if you'd like me to have a look :-)
Ghis
building manpages via setup.py
Hello,
I am working on the pyfai package.
This pacakge contain one module with extensions (the important point)
The new upstream version 0.14.0 provide a build_man target via the setup.py
So in ordert to generate the doc I need to do
python setup.py build_man
Now if I look at this target, I can find this code
-
class BuildMan(Command):
"""Command to build man pages"""
user_options = []
def initialize_options(self):
pass
def finalize_options(self):
pass
def entry_points_iterator(self):
"""Iterate other entry points available on the project."""
entry_points = self.distribution.entry_points
console_scripts = entry_points.get('console_scripts', [])
gui_scripts = entry_points.get('gui_scripts', [])
scripts = []
scripts.extend(console_scripts)
scripts.extend(gui_scripts)
for script in scripts:
elements = script.split("=")
target_name = elements[0].strip()
elements = elements[1].split(":")
module_name = elements[0].strip()
function_name = elements[1].strip()
yield target_name, module_name, function_name
def run(self):
build = self.get_finalized_command('build')
path = sys.path
path.insert(0, os.path.abspath(build.build_lib))
env = dict((str(k), str(v)) for k, v in os.environ.items())
env["PYTHONPATH"] = os.pathsep.join(path)
import subprocess
status = subprocess.call(["mkdir", "-p", "build/man"])
if status != 0:
raise RuntimeError("Fail to create build/man directory")
import tempfile
import stat
script_name = None
entry_points = self.entry_points_iterator()
for target_name, module_name, function_name in entry_points:
logger.info("Build man for entry-point target '%s'" % target_name)
# help2man expect a single executable file to extract the help
# we create it, execute it, and delete it at the end
py3 = sys.version_info >= (3, 0)
try:
# create a launcher using the right python interpreter
script_fid, script_name = tempfile.mkstemp(prefix="%s_" %
target_name, text=True)
script = os.fdopen(script_fid, 'wt')
script.write("#!%s\n" % sys.executable)
script.write("import %s as app\n" % module_name)
script.write("app.%s()\n" % function_name)
script.close()
# make it executable
mode = os.stat(script_name).st_mode
os.chmod(script_name, mode + stat.S_IEXEC)
# execute help2man
man_file = "build/man/%s.1" % target_name
command_line = ["help2man", script_name, "-o", man_file]
if not py3:
# Before Python 3.4, ArgParser --version was using
# stderr to print the version
command_line.append("--no-discard-stderr")
p = subprocess.Popen(command_line, env=env)
status = p.wait()
if status != 0:
raise RuntimeError("Fail to generate '%s' man
documentation" % target_name)
finally:
# clean up the script
if script_name is not None:
os.remove(script_name)
-
As you can see this create a launch script for each entry point found in the
setup.py and run help2man on it.
For now I would like to use this setup.py without modification.
So what should I do to run
python setup.py build_man with the options provided by pybuild during the
normal build in order to let the generated script find the pyFAI modules and
its extensions ?
Second questions what is the right way to generate the man pages for a python
application ?
thanks for your help
Frederic
