Re: packaging problems with python3.5 and python 3.6
On Wed, Aug 02, 2017 at 12:10:05PM -0400, ba...@debian.org wrote: > On Jul 30, 2017, at 02:41, Steve Langasek wrote: > >> https://wiki.debian.org/Python/LibraryStyleGuide where it states: > >>You'll want to have at least the following build dependencies: > >>... > >>python3-all > >> but the > >>python[,3]-all-dev > >> are never mentioned. So sorry, I didn't know about them and searched > >> for the required header files, and added the respective packages. > > Ok, that was the first hit I got but I discounted it because it spoke of > > packaging "libraries", not modules or extensions. > “Libraries” is typically the term we’ve used to describe things you import > from other things. In Python, everything you import is a module, and > packages are just special kinds of modules, whereas extensions are module > implemented in a compiled language instead of native Python. It may not be > the best nomenclature, but it’s what we’ve been using for a long time. > > Perhaps someone should make this wiki page a redirect to the python policy > > manual? https://www.debian.org/doc/packaging-manuals/python-policy/ > No, because LibraryStyleGuide *isn’t* policy, it’s an opinionated > collection of best practices, and it is better suited for a wiki because > those best practices do change over time. > Policy is what you MUST do, and the barrier for changing policy is — and > needs to be — higher. Except the python packaging policy already mentions build dependencies: "The python3-all-dev should be used when building extensions for any or all Python 3 versions. The python-all-dev should be used when building extensions for any or all Python 2 versions." https://www.debian.org/doc/packaging-manuals/python-policy/ap-build_dependencies.html But this is then shadowed by the less precise/complete recommendation in the wiki. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature
Re: packaging problems with python3.5 and python 3.6
On Jul 30, 2017, at 02:41, Steve Langasek wrote: >> https://wiki.debian.org/Python/LibraryStyleGuide where it states: >> You'll want to have at least the following build dependencies: >> ... >> python3-all >> but the >> python[,3]-all-dev >> are never mentioned. So sorry, I didn't know about them and searched >> for the required header files, and added the respective packages. > Ok, that was the first hit I got but I discounted it because it spoke of > packaging "libraries", not modules or extensions. “Libraries” is typically the term we’ve used to describe things you import from other things. In Python, everything you import is a module, and packages are just special kinds of modules, whereas extensions are module implemented in a compiled language instead of native Python. It may not be the best nomenclature, but it’s what we’ve been using for a long time. > Perhaps someone should make this wiki page a redirect to the python policy > manual? https://www.debian.org/doc/packaging-manuals/python-policy/ No, because LibraryStyleGuide *isn’t* policy, it’s an opinionated collection of best practices, and it is better suited for a wiki because those best practices do change over time. Policy is what you MUST do, and the barrier for changing policy is — and needs to be — higher. Cheers, -Barry signature.asc Description: Message signed with OpenPGP
RE:building manpages via setup.py
> The snippet you quoted is not specific to extension modules but to the > use of the autodoc feature, which requires the modules to be in the > PYTHONPATH. The `sys.path.insert` hack is just here so that you don't > have to specify PYTHONPATH yourself when running the upstream Makefile. I just says that this snipset is in the conf.py and I yes we need this for autodoc. > I don't understand how setup.py / build options are affecting the Sphinx > documentation. You are supposed to either call the generator manually > via sphinx-build (which the style guide recommends), or using the > upstream Makefile (which upstream often does). autodoc introspect the module so you need a working modules from python, whcih is avhieved via the right PYTHONPATH. So, during the build phase you can gives options on the python setup.py command line. This has an effect on the location where the module and the extensions are built. build seems to be the default location BUT when we use pybuild it is under .pybuild So the default PYTHONPATH does not work. there is no way for sphinx to know where the module and its extensions are excepty by providing the right PYTHONPATH. Except if sphinx would provide a build_sphinx command whcih take care of this problem and take into account the parameters of the setup.py. > As far as reliability is concerned, my reference here is upstream. If > upstream can produce the docs, then I should be able to do it too. And > they don't have pybuild for that. Yes but sometime there si some hack done by the upstream in conf.py or setup.py in order to look for the module at the standards path (under build/) which doesn not work with pybuild. > Why would you build the docs for each supported Python version, > considering you'll end up shipping only one instance of the generated > HTML? I am probably missing something here. I use this as a sort of unit test. And it is interesting because I catche a bunch of python2/python3 issues like this. Then I can find issues between sphinx running on python2 or python3. This is at least a good way to check thaht all the modules can be laoded. autodoc is interesting for this. > You might as well generate the docs just once with the default Python 2 > or Python 3 interpreter, the same way you would do without extension > modules, no? > Pardon my insistence, but I really fail to understand what issues you > are referring to. incompatibilities vetween python2 and python 3 code. peoples trie to write code python2 and python3 compatible but this is not thaht easy... > More than I expected, indeed. > And amongst these 643 packages, how many have such large build times > that the overhead of an additional inplace call would be considered > prohibitive, I wonder? I do not know, I just think that we should be able to automatically generate packages for python modules from pip in order to reduce the effort of pacakging, So there should be only one way to build the documentation , manpages. Escpecillay wheren there is extensions. > Besides, the docs will typically be processed by arch-all builders > (provided you use -indep targets), so arch-any builds won't even feel > that overhead, right? Yes except that this is ot because --inplace work that the result of python setup.py build are identical. it is possible to forget a module referenced by another module in the setup.py. It workd with --inplace but it does not work for real after build ans installation. Cheers Fred I could be wrong though.
Re: building manpages via setup.py
On 02/08/17 10:45, PICCA Frederic-Emmanuel wrote: First, that's very speculative. Second, that's upstream's problem. # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # # import os # import sys # sys.path.insert(0, os.path.abspath('.')) You need to customize the sys.path in order to find the extensions. The snippet you quoted is not specific to extension modules but to the use of the autodoc feature, which requires the modules to be in the PYTHONPATH. The `sys.path.insert` hack is just here so that you don't have to specify PYTHONPATH yourself when running the upstream Makefile. This path can change depending on the setup.py build options so this is not reliable. I don't understand how setup.py / build options are affecting the Sphinx documentation. You are supposed to either call the generator manually via sphinx-build (which the style guide recommends), or using the upstream Makefile (which upstream often does). As far as reliability is concerned, my reference here is upstream. If upstream can produce the docs, then I should be able to do it too. And they don't have pybuild for that. Why? All you need is *one* occurrence of the extension modules somewhere in the PYTHONPATH in order to generate the docs. Chances are that's how upstream generates them. Because by experience I find issue in the build system and the python code when building the doc for multiple interpreter (python2/python3 differences...) Why would you build the docs for each supported Python version, considering you'll end up shipping only one instance of the generated HTML? I am probably missing something here. You might as well generate the docs just once with the default Python 2 or Python 3 interpreter, the same way you would do without extension modules, no? Pardon my insistence, but I really fail to understand what issues you are referring to. Found a total of 643 reverse build-depend(s) for python-all-dev. not that small How is the ratio over all the Python packages? I suspect very small. Found a total of 1968 reverse build-depend(s) for python-all. not that small 32 % ;) More than I expected, indeed. And amongst these 643 packages, how many have such large build times that the overhead of an additional inplace call would be considered prohibitive, I wonder? Besides, the docs will typically be processed by arch-all builders (provided you use -indep targets), so arch-any builds won't even feel that overhead, right? I could be wrong though.
RE:building manpages via setup.py
> First, that's very speculative. Second, that's upstream's problem. > The upstream Makefile and conf.py are likely generated by Sphinx itself > via sphinx-quickstart. Did your upstream tinker with them that much that > they cannot be trusted? No this is just that does not fit well with extension. > Some upstream do use a build_sphinx command, but it is far from common > and it does not solve the extension module problem. Yes this is what they are doing. extension are part of the python ec-system but it seems thaht this is not well suported out of the box by sphinx. # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # # import os # import sys # sys.path.insert(0, os.path.abspath('.')) You need to customize the sys.path in order to find the extensions. This path can change depending on the setup.py build options so this is not reliable. > Why? All you need is *one* occurrence of the extension modules somewhere > in the PYTHONPATH in order to generate the docs. Chances are that's how > upstream generates them. Because by experience I find issue in the build system and the python code when building the doc for multiple interpreter (python2/python3 differences...) > > Found a total of 643 reverse build-depend(s) for python-all-dev. > > > > not that small > How is the ratio over all the Python packages? I suspect very small. Found a total of 1968 reverse build-depend(s) for python-all. not that small 32 % ;) Cheers Fred
Re: building manpages via setup.py
On 02/08/17 09:55, PICCA Frederic-Emmanuel wrote: PYTHONPATH=. sphinx-build -N -b html One can also use the sphinx-generated Makefile if available: PYTHONPATH=$(CURDIR) $(MAKE) -C html Both are simple one-liners and do not rely on pybuild. Yes it works but this is fragile since the organisation of the module can change in the sources. First, that's very speculative. Second, that's upstream's problem. I have not seen many upstream projects playing with the layout of their modules from one version to the next. If that's the case, then there are worst things to be worried about (API breakage for instance) than the docs. at least the .pybuild directory is under the responsability of pybuild and we should use pybuild instead of relying on the maintainer snipset. (typo error, change during the time.) The upstream Makefile and conf.py are likely generated by Sphinx itself via sphinx-quickstart. Did your upstream tinker with them that much that they cannot be trusted? It would be nice if the doc generation in python shold be standardize. Some upstream do use a build_sphinx command, but it is far from common and it does not solve the extension module problem. If it does not cost much to build the extension inplace, then the simplest option is to prefix one of these calls above with: python3 setup.py build_ext --inplace when you have multiple verison of the interpreter you prefer to avoid --inplace. Why? All you need is *one* occurrence of the extension modules somewhere in the PYTHONPATH in order to generate the docs. Chances are that's how upstream generates them. I fail to picture how this is an issue in practice considering your build override will run the isolated pybuilds first and the sphinx call last. If the cost is prohibitive, which arguably applies to a very limited set of packages (yours included) then you would use pybuild for that, as Piotr kindly suggested. Yes it depends olsa of the arch. Some are really slow That's unfortunate, indeed. Considering the rarity of this use case though, I wonder whether it is worth adding a separate section to the style guide. Found a total of 643 reverse build-depend(s) for python-all-dev. not that small How is the ratio over all the Python packages? I suspect very small.
RE:building manpages via setup.py
> PYTHONPATH=. sphinx-build -N -b html > One can also use the sphinx-generated Makefile if available: > PYTHONPATH=$(CURDIR) $(MAKE) -C html > Both are simple one-liners and do not rely on pybuild. Yes it works but this is fragile since the organisation of the module can change in the sources. at least the .pybuild directory is under the responsability of pybuild and we should use pybuild instead of relying on the maintainer snipset. (typo error, change during the time.) It would be nice if the doc generation in python shold be standardize. > If it does not cost much to build the extension inplace, then the > simplest option is to prefix one of these calls above with: > python3 setup.py build_ext --inplace when you have multiple verison of the interpreter you prefer to avoid --inplace. > If the cost is prohibitive, which arguably applies to a very limited set > of packages (yours included) then you would use pybuild for that, as > Piotr kindly suggested. Yes it depends olsa of the arch. Some are really slow > Considering the rarity of this use case though, I wonder whether it is > worth adding a separate section to the style guide. Found a total of 643 reverse build-depend(s) for python-all-dev. not that small cheers Fred
RE:building manpages via setup.py
> At the end of the day, it is just a matter of providing an appropriate > PYTHONPATH, regardless of whether pybuild is used or not. Yes but to avoid the multiplications of way to provide this PYTHONPATH. Is it possible to have the recommended way which works for modules and extensions. once agreed, we should put this in the wiki Cheers Fred
Re: building manpages via setup.py
On 02/08/17 09:19, PICCA Frederic-Emmanuel wrote: At the end of the day, it is just a matter of providing an appropriate PYTHONPATH, regardless of whether pybuild is used or not. Yes but to avoid the multiplications of way to provide this PYTHONPATH. For the vast majority of packages, the current method listed in LibraryStyleGuide applies, i.e.: PYTHONPATH=. sphinx-build -N -b html One can also use the sphinx-generated Makefile if available: PYTHONPATH=$(CURDIR) $(MAKE) -C html Both are simple one-liners and do not rely on pybuild. Is it possible to have the recommended way which works for modules and extensions. If it does not cost much to build the extension inplace, then the simplest option is to prefix one of these calls above with: python3 setup.py build_ext --inplace If the cost is prohibitive, which arguably applies to a very limited set of packages (yours included) then you would use pybuild for that, as Piotr kindly suggested. once agreed, we should put this in the wiki Considering the rarity of this use case though, I wonder whether it is worth adding a separate section to the style guide. Ghis
RE:building manpages via setup.py
> Perhaps the LibraryStyleGuide should be updated to reflect on this > change? I believe we are still advising explicit http_proxy / > https_proxy exports prior to running sphinx-build. And running sphinx-build does not work expecially if there is extensions in the documentation. sphinx-build should be run via pybuild in order to know about the build_dir. right ? Fred
Re: building manpages via setup.py
On 02/08/17 09:03, PICCA Frederic-Emmanuel wrote: Perhaps the LibraryStyleGuide should be updated to reflect on this change? I believe we are still advising explicit http_proxy / https_proxy exports prior to running sphinx-build. And running sphinx-build does not work expecially if there is extensions in the documentation. sphinx-build should be run via pybuild in order to know about the build_dir. right ? At the end of the day, it is just a matter of providing an appropriate PYTHONPATH, regardless of whether pybuild is used or not. The default is $(CURDIR) for the vast majority of packages. But, it may also be something else if extension packages are involved, or if the package directory is under a different folder than the root directory, such as src/. Ghis
Re: building manpages via setup.py
On 02/08/17 08:44, Piotr Ożarowski wrote: [PICCA Frederic-Emmanuel, 2017-08-02] you can drop it, PYTHONPATH and http_proxy should be set by pybuild Is it true for jessie I need to support jessie and stretch And even debian7... I didn't test it even for unstable, but IIRC pybuild exports those in all steps since a long time ago. You'll know after first builds... Perhaps the LibraryStyleGuide should be updated to reflect on this change? I believe we are still advising explicit http_proxy / https_proxy exports prior to running sphinx-build. Ghis
RE:building manpages via setup.py
> export PYBUILD_AFTER_BUILD={interpreter} setup.py build_man Thanks, The only drawback I see with this solution is that I want to run dh_auto_build 2 times, - 1 for the arch part (pyhon modules, extensions and manpages) - 2 for the indep part (doc build) nevertheless thans a lot it is flexxible enought for my case. > you can drop it, PYTHONPATH and http_proxy should be set by pybuild Is it true for jessie I need to support jessie and stretch And even debian7... Fred
Re: building manpages via setup.py
[PICCA Frederic-Emmanuel, 2017-08-02] > > you can drop it, PYTHONPATH and http_proxy should be set by pybuild > > Is it true for jessie > > I need to support jessie and stretch > > And even debian7... I didn't test it even for unstable, but IIRC pybuild exports those in all steps since a long time ago. You'll know after first builds... PS please don't CC me, I'm subscribing the list -- GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: updating packages
On 2017-08-01 11:58, Christopher Hoskin wrote: > What's the plan for moving them to unstable? Are they still using git > pq rather than git-dpm? No plan yet for an upload to unstable. amqp and kombu have quite a few reverse dependencies and I did no have the time yet to try a rebuild on them. All update packages have been converted to gbp pq. Cheers, -- Michael Fladischer Fladi.at signature.asc Description: OpenPGP digital signature
Re: building manpages via setup.py
[PICCA Frederic-Emmanuel, 2017-08-02] > > if you want to test all requested Python interpreters: > > I prefer this solution in order to check that the built extensions are > working for all the python interpreters. > The doc use autodoc so it is nice to have this functionnality > > | override_dh_auto_build: > | dh_auto_build -- --after-build '{interpreter} setup.py build_man' > > > It is possible to replace this by > > export PYBUILD_AFTER_BUILD='{interpreter} setup.py build_man' export PYBUILD_AFTER_BUILD={interpreter} setup.py build_man (i.e. without quotes if you want to use Make's export) > > there's no need to prepend --after-build argument with > > "env PYTHONPATH={build_dir}; " - pybuild sets that to build dir > > automatically > > Do you mean that I should remove the PYTHONPATH in this ? you can drop it, PYTHONPATH and http_proxy should be set by pybuild > override_dh_sphinxdoc > ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS))) > PYBUILD_SYSTEM=custom \ > PYBUILD_BUILD_ARGS="cd doc && PYTHONPATH={build_dir} > http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source build/html" > dh_auto_build # HTML generator > dh_installdocs "doc/build/html" -p python-pyfai-doc > dh_sphinxdoc -O--buildsystem=pybuild > endif -- GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
RE:building manpages via setup.py
Hello piotr > if you want to test all requested Python interpreters: I prefer this solution in order to check that the built extensions are working for all the python interpreters. The doc use autodoc so it is nice to have this functionnality | override_dh_auto_build: | dh_auto_build -- --after-build '{interpreter} setup.py build_man' It is possible to replace this by export PYBUILD_AFTER_BUILD='{interpreter} setup.py build_man' > there's no need to prepend --after-build argument with > "env PYTHONPATH={build_dir}; " - pybuild sets that to build dir automatically Do you mean that I should remove the PYTHONPATH in this ? override_dh_sphinxdoc ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS))) PYBUILD_SYSTEM=custom \ PYBUILD_BUILD_ARGS="cd doc && PYTHONPATH={build_dir} http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source build/html" dh_auto_build # HTML generator dh_installdocs "doc/build/html" -p python-pyfai-doc dh_sphinxdoc -O--buildsystem=pybuild endif -- GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645