Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Dmitry Shachnev]

Hi Brian,

On July 17, 2016 3:44:29 AM GMT+02:00, Brian May  wrote:
> Dmitry Shachnev  writes:
> 
> > Setuptools provides a build_sphinx command out of the box. For
> packages
> > using distutils, there is a sphinx.setup_command module provided by
> > Sphinx.
> >
> > Some upstreams call that command during build phase, though that’s
> not
> > common.
> 
> Is there any reason that this isn't more common? Should I be doing
> this
> for my packages which I maintain upstream?

If you are both the upstream and the packager, then adding a few lines in 
debian/rules to build the documentation is probably easier than hacking the 
build command in setup.py.

--
Dmitry Shachnev

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Brian May]

Dmitry Shachnev  writes:

> Setuptools provides a build_sphinx command out of the box. For packages
> using distutils, there is a sphinx.setup_command module provided by
> Sphinx.
>
> Some upstreams call that command during build phase, though that’s not
> common.

Is there any reason that this isn't more common? Should I be doing this
for my packages which I maintain upstream?
-- 
Brian May 

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Piotr Ożarowski]

[Andreas Tille, 2016-07-14]
> override_dh_auto_build:
> # arch
> USE_CYTHON=true dh_auto_build
> # indep:

override_dh_auto_build-indep:

> PYTHONPATH=. http_proxy='127.0.0.1:9' sphinx-build -N -b html doc 
> build/html
> PYTHONPATH=. http_proxy='127.0.0.1:9' sphinx-build -N -b man doc 
> build/man
> 

(python-sphinx is only in Build-Depends-Indep hence you cannot use it in
override_dh_auto_build)
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Sergio Durigan Junior]

On Thursday, July 14 2016, Andreas Tille wrote:

>> Not sure I understand how dh-python / pybuild help with this problem.
>> 
>> I thought the recommended way to deal with building documentation was
>> something like this in debian/rules:
>> 
>> override_dh_auto_build:
>> dh_auto_build
>> PYTHONPATH=. sphinx-build -b html -N Doc/ Doc/.build/html
>> 
>> ... in which case building the documentation happens outside
>> dh_auto_build.
>
> I'm using
>
> override_dh_auto_build:
> # arch
> USE_CYTHON=true dh_auto_build
> # indep:
> PYTHONPATH=. http_proxy='127.0.0.1:9' sphinx-build -N -b html doc 
> build/html
> PYTHONPATH=. http_proxy='127.0.0.1:9' sphinx-build -N -b man doc 
> build/man
>
> in python-biom-format[1] which results in #831239.  I was once advised
> to specify the http_proxy that way (others here in this thread as well)
> and I wonder how to solve this.  If I try to export http_proxy and do
> not call sphinx-build manually at all no documentation will be created.

Hey Andreas,

You have to instruct debhelper to use dh_sphinxdoc.  I.e.:

  dh $@ --with python2,python3,bash-completion,sphinxdoc --buildsystem=pybuild

Cheers,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Andreas Tille]

Hi,

On Sun, Jul 10, 2016 at 11:34:53AM +1000, Brian May wrote:
> Moved to debian-python list from bug report.

Great - so I can step in here with an equivalent problem. :-)
 
> Not sure I understand how dh-python / pybuild help with this problem.
> 
> I thought the recommended way to deal with building documentation was
> something like this in debian/rules:
> 
> override_dh_auto_build:
> dh_auto_build
> PYTHONPATH=. sphinx-build -b html -N Doc/ Doc/.build/html
> 
> ... in which case building the documentation happens outside
> dh_auto_build.

I'm using

override_dh_auto_build:
# arch
USE_CYTHON=true dh_auto_build
# indep:
PYTHONPATH=. http_proxy='127.0.0.1:9' sphinx-build -N -b html doc 
build/html
PYTHONPATH=. http_proxy='127.0.0.1:9' sphinx-build -N -b man doc 
build/man

in python-biom-format[1] which results in #831239.  I was once advised
to specify the http_proxy that way (others here in this thread as well)
and I wonder how to solve this.  If I try to export http_proxy and do
not call sphinx-build manually at all no documentation will be created.

So how to solve this?

Kind regards

   Andreas.

[1] https://anonscm.debian.org/git/debian-med/python-biom-format.git

-- 
http://fam-tille.de

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Barry Warsaw]

On Jul 12, 2016, at 04:07 PM, Thomas Goirand wrote:

>I find this not the top-noch way. Here's what I do:
>
>override_dh_sphinxdoc:
>ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
>   PYTHONPATH=. sphinx-build -b html doc/source \
>   debian/python-foo-doc/usr/share/doc/python-foo-doc/html
>   dh_sphinxdoc
>endif

I'm going to do this in the next python-webob upload to close #830611.  It had
been building the docs in override_dh_installdocs.  I'll also need to patch
the conf.py file to use the python{,3}-doc files and add those packages as
build dependencies.

Cheers,
-Barry


pgpROt_11jI7v.pgp
Description: OpenPGP digital signature

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Thomas Goirand]

On 07/10/2016 03:34 AM, Brian May wrote:
> Moved to debian-python list from bug report.
> 
> Dmitry Shachnev  writes:
> 
>> You say that we can read some flag set by dh-python. However, dh-python
>> (aka pybuild) already sets http{,s}_proxy environment variables which
>> disable network access for Sphinx. So packages using dh-python are already
>> safe.
> 
> Not sure I understand how dh-python / pybuild help with this problem.

It doesn't, it simply hides the issue.

Also, sometimes exporting http{s,}_proxy simply breaks unit testing (I
did have such case, yes...). So that's not really a fit-all solution either.

> I thought the recommended way to deal with building documentation was
> something like this in debian/rules:
> 
> override_dh_auto_build:
> dh_auto_build
> PYTHONPATH=. sphinx-build -b html -N Doc/ Doc/.build/html

I find this not the top-noch way. Here's what I do:

override_dh_sphinxdoc:
ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
PYTHONPATH=. sphinx-build -b html doc/source \
debian/python-foo-doc/usr/share/doc/python-foo-doc/html
dh_sphinxdoc
endif

IMO, that's better because you don't need to clean Doc/.build/html.
Also, the dh_auto_build isn't the correct target. Some derivative may
well choose not to implement dh_sphinxdoc (for example, for small
systems where they don't want docs), and here, you're making their lives
miserable.

Cheers,

Thomas Goirand (zigo)

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Dmitry Shachnev]

Hi Brian,

On Tue, Jul 12, 2016 at 07:45:01AM +1000, Brian May wrote:
> How do you call Sphinx in setup.py? When I researched this before, the
> best advice I could find seemed to be along the lines of "don't do
> that".

Setuptools provides a build_sphinx command out of the box. For packages
using distutils, there is a sphinx.setup_command module provided by
Sphinx.

Some upstreams call that command during build phase, though that’s not
common.

I agree that my comment was a bit confusing because it applies only to
this rare case (it was mainly an answer to Chris’ question which mentioned
pybuild).

-- 
Dmitry Shachnev


signature.asc
Description: PGP signature

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Brian May]

Dmitry Shachnev  writes:

> Indeed my comment applies only if Sphinx is invoked by pybuild (for example
> if it's called by setup.py build).

How do you call Sphinx in setup.py? When I researched this before, the
best advice I could find seemed to be along the lines of "don't do
that".
-- 
Brian May 

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Daniele Tricoli]

Hello Dmitry,

On Monday, July 11, 2016 09:06:57 PM Dmitry Shachnev wrote:
> If it's invoked separately the only ways would be to export http(s)_proxy in
> debian/rules or patch conf.py to get correct intersphinx mapping.

Thanks for the details.
I'm already exporting http_proxy inside betamax debian/rules, maybe patch 
conf.py is needed anyway due DNS queries: it seems confirmed looking at 
tcpdump log for #830543.

Thanks! Cheers,

-- 
 Daniele Tricoli 'eriol'
 https://mornie.org

signature.asc
Description: This is a digitally signed message part.

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Dmitry Shachnev]

Hi Brian,

On Sun, Jul 10, 2016 at 11:34:53AM +1000, Brian May wrote:
> Not sure I understand how dh-python / pybuild help with this problem.
>
> I thought the recommended way to deal with building documentation was
> something like this in debian/rules:
>
> override_dh_auto_build:
> dh_auto_build
> PYTHONPATH=. sphinx-build -b html -N Doc/ Doc/.build/html
>
> ... in which case building the documentation happens outside
> dh_auto_build.
>
> Did I miss something here?

Indeed my comment applies only if Sphinx is invoked by pybuild (for example
if it's called by setup.py build).

If it's invoked separately the only ways would be to export http(s)_proxy in
debian/rules or patch conf.py to get correct intersphinx mapping.

Unfortunately there is no easy way for Sphinx to detect if it's invoked as
part of the package build process, so the ways I can change it are limited.

-- 
Dmitry Shachnev


signature.asc
Description: PGP signature

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Daniele Tricoli]

Hello Brian,

On Sunday, July 10, 2016 11:34:53 AM Brian May wrote:
> ... in which case building the documentation happens outside
> dh_auto_build.
> 
> Did I miss something here?

I have the same issue with betamax, and IIUC #830186 we have to patch 
intersphinx to use the distro provided inventory files. See this reply of 
Dmitry Shachnev (at the end he point to few examples):

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=830186#10

Cheers,

-- 
 Daniele Tricoli 'eriol'
 https://mornie.org

signature.asc
Description: This is a digitally signed message part.

Re: [Python-modules-team] Bug#830186: sphinx: intersphinx mapping extension causes network access during package builds

[Brian May]

Moved to debian-python list from bug report.

Dmitry Shachnev  writes:

> You say that we can read some flag set by dh-python. However, dh-python
> (aka pybuild) already sets http{,s}_proxy environment variables which
> disable network access for Sphinx. So packages using dh-python are already
> safe.

Not sure I understand how dh-python / pybuild help with this problem.

I thought the recommended way to deal with building documentation was
something like this in debian/rules:

override_dh_auto_build:
dh_auto_build
PYTHONPATH=. sphinx-build -b html -N Doc/ Doc/.build/html

... in which case building the documentation happens outside
dh_auto_build.

Did I miss something here?
-- 
Brian May