Bug#961206: improve sphinx usage for cross building

[Dmitry Shachnev]

Hi Helmut!

On Fri, Jun 12, 2020 at 01:02:26PM +0200, Helmut Grohne wrote:
> Hi Dmitry,
>
> > So what are our next steps? I will make a Sphinx 3.x upload with updated
> > package description to experimental soon.
>
> Unfortunately, I encountered another little roadblock. At least
> sphinx-apidoc imports modules. Doing so certainly isn't M-A:foreign. I
> don't know whether sphinx-autogen also does. In any case, this doesn't
> work as planned.

Ah, yes. One use of Sphinx is automatically generating documentation from
Python code, and it definitely requires importing code.

sphinx.ext.autodoc is one place where it happens, but some other extensions
(like sphinx.ext.coverage and sphinx.ext.inheritance_diagram) also do it.
Grepping the code for import_module gives an approximation of where else
this is used.

At least autodoc is very popular:
https://codesearch.debian.net/search?q=sphinx.ext.autodoc

I just thought that maybe this code could be moved to a separate package
(not M-A: foreign), but that would require a (permanent) circular dependency,
as these extensions use code from Sphinx core.

> Given that I've performed the analysis under the assumption that marking
> sphinx M-A:foreign was ok, the rest of this email is going to assume
> that even though this assumption is now known to be wrong.
>
> We could do with a temporary, circular dependency. We could split the
> sphinx package from python3-sphinx right now. Now sphinx must depend on
> python3-sphinx forever and python3-sphinx must depend on sphinx
> temporarily to avoid breaking rdeps. The latter dependency can be
> dropped in like half a year maybe.
>
> As soon as this split is in place, we can mark sphinx Multi-Arch:
> foreign (assuming this was fine). Once that happens, cross builds can
> immediately benefit from the split without requiring that everyone else
> updates their Build-Depends.

So the tricky part here is “assuming this was fine”. If that can be assumed,
then having a temporary circular dependency is OK for me.

> > Do you want to prepare mass bug filing, or you prefer that we deal with the
> > packages in Python team (DPMT/PAPT) first?
>
> I strongly prefer reducing the number of bugs to file before filing
> them.

I agree.

> There are also two parts of this transition:
>
>  A) Fixing packages to work without the python3-sphinx -> sphinx
> dependency.
>  B) Fixingg packages to use sphinx instead of python3-sphinx for cross
> building.
>
> I think B is a long-tail transition and you basically don't have to care
> about it at all. A is the transition you need to pay attention to.
>
> So how about numbers?
> 
> Around 1200 packages (transivitively) build depend on python3-sphinx. I
> built them with a path-exclude for the sphinx tools.  Around 500
> packages FTBFS when /usr/bin/sphinx-* is removed. Around 450 clearly
> fail at using one of the tools. The rest is roughly 25 filed FTBFS and
> 25 packages where I have no clue.
>
> Here is the list of packages that definitely use a sphinx commandline
> tool:
> [...]

So you already have a list of packages that need bugs, so if we decide to
proceed you can just file them in a bulk, asking to add sphinx
build-dependency (and possibly remove python3-sphinx).

> I'm quite unsure about how to proceed from here. The split doesn't work
> as planned. :-/

Same for me…

--
Dmitry Shachnev


signature.asc
Description: PGP signature

Bug#961206: improve sphinx usage for cross building

[Helmut Grohne]

Hi Dmitry,

On Tue, Jun 02, 2020 at 01:03:15PM +0300, Dmitry Shachnev wrote:
> [ Adding the bug back to Cc, I assume you did not intend to remove it. ]

Thank you. That was unintentional indeed. Fortunately, you quoted
everything.

> So what are our next steps? I will make a Sphinx 3.x upload with updated
> package description to experimental soon.

Unfortunately, I encountered another little roadblock. At least
sphinx-apidoc imports modules. Doing so certainly isn't M-A:foreign. I
don't know whether sphinx-autogen also does. In any case, this doesn't
work as planned.

Given that I've performed the analysis under the assumption that marking
sphinx M-A:foreign was ok, the rest of this email is going to assume
that even though this assumption is now known to be wrong.

We could do with a temporary, circular dependency. We could split the
sphinx package from python3-sphinx right now. Now sphinx must depend on
python3-sphinx forever and python3-sphinx must depend on sphinx
temporarily to avoid breaking rdeps. The latter dependency can be
dropped in like half a year maybe.

As soon as this split is in place, we can mark sphinx Multi-Arch:
foreign (assuming this was fine). Once that happens, cross builds can
immediately benefit from the split without requiring that everyone else
updates their Build-Depends.

> Do you want to prepare mass bug filing, or you prefer that we deal with the
> packages in Python team (DPMT/PAPT) first?

I strongly prefer reducing the number of bugs to file before filing
them.

There are also two parts of this transition:

 A) Fixing packages to work without the python3-sphinx -> sphinx
dependency.
 B) Fixingg packages to use sphinx instead of python3-sphinx for cross
building.

I think B is a long-tail transition and you basically don't have to care
about it at all. A is the transition you need to pay attention to.

So how about numbers?

Around 1200 packages (transivitively) build depend on python3-sphinx. I
built them with a path-exclude for the sphinx tools.  Around 500
packages FTBFS when /usr/bin/sphinx-* is removed. Around 450 clearly
fail at using one of the tools. The rest is roughly 25 filed FTBFS and
25 packages where I have no clue.

Here is the list of packages that definitely use a sphinx commandline
tool:

ahven aiohttp-wsgi aiomysql alot ansible astroplan astroquery
azure-cosmos-python azure-cosmos-table-python
azure-data-lake-store-python basemap bcolz beanbag bedops beets
bibtexparser binoculars blends breathe breezy bugwarrior buildbot casync
cdist ceph ceres-solver charliecloud cheetah chemps2 class.js cmake
compass-susy-plugin configobj configparser cpl-plugin-amber
cpl-plugin-fors cpl-plugin-giraf cpl-plugin-hawki cpl-plugin-muse
cpl-plugin-naco cpl-plugin-uves cpl-plugin-vimos cpl-plugin-visir
cpl-plugin-xshoo csvkit cvxopt cysignals cython dask dballe dcm2niix
deap debian-policy debomatic defcon developers-reference dex dhcpcanon
diceware django-axes django-guardian django-pipeline dms dogtag-pki doit
dolfin dolfinx dpdk dput-ng dune-common dynare elastalert epigrass
fabric fastchunking ffcx fityk flask-bcrypt flask-flatpages
flask-restful flask-wtf flent flycheck fmtlib fontparts fonttools fsspec
gcc-python-plugin gearmand ghc git-cola glom gnat-gps goiardi
goocalendar grapefruit graphite-api gromacs guidata gwcs h5py
hamradio-maintguide haproxy hdf-compass heat-cfntools hiera-py
highlight.js hy hyperkitty igdiscover isbg isc-kea isso itango jansson
jeepney jinja2 jsonpickle khard kitty knot knot-resolver krb5 lammps
lasagne libabigail libaws libcbor libcork libcorkipset libgnatcoll
libgpuarray libixion liblognorm liborcus libserial libtemplates-parser
libvigraimpex libxmlada llvm-toolchain-10 llvm-toolchain-8
llvm-toolchain-9 llvmlite logbook loggerhead logilab-common logzero
lua-argparse lua-unit luabind luacheck macsyfinder mailman3
mailmanclient mako mame mapproxy mash mathjax-docs
microsoft-authentication-library-for-python minieigen mlpy
mongo-c-driver morse-simulator mpd mpi4py-fft mpmath musicbrainzngs
mydumper mypy ndcube neo networkx nextcloud-desktop nfs-ganesha nghttp2
nitime nordugrid-arc-nagios-plugins notmuch ns3 ntopng numba numcodecs
numpy oar objgraph ocrmypdf onionbalance opencolorio openvswitch
opgpcard owncloud-client owslib pagure pam-python pandas parso
pathspider patroni patsy pdal pelican pgloader pgrouting photofilmstrip
php-mockery php-twig phpmyadmin pikepdf pillow pocl podcastparser
polybar postorius powerline pplpy praw psycopg2 py3c pybind11 pybindgen
pybtex pybtex-docutils pycares pycoast pycollada pydicom pydl pydocstyle
pyepr pyfftw pyfr pygccjit pygccxml pygments pygresql pyhamtools pyliblo
pylint pymongo pyopencl pyopenssl pyosmium pyparsing pypy pypy3 pyqi
pyqso pyregion pyresample pyrlp pyro4 pysdl2 pyserial-asyncio
pysoundfile pyswarms pytables pytest pytest-mpi pytest-qt python-acme
python-agate python-agate-dbf python-agate-excel python-agate-sql
python-aiohttp-security python-aiohttp-session python-aioice

Bug#961206: improve sphinx usage for cross building

[Dmitry Shachnev]

Hi Helmut!

[ Adding the bug back to Cc, I assume you did not intend to remove it. ]

On Thu, May 28, 2020 at 03:25:08PM +0200, Helmut Grohne wrote:
> I think we can slightly simplify it:
>
> | Build-depend on sphinx if your package uses /usr/bin/sphinx-*
> | executables. Build-depend on python3-sphinx if your package uses the
> | Python API (for instance by calling python3 -m sphinx).
>
> In some cases that may result in requiring both dependencies.

That sounds fine, I added these words to the package description.

https://salsa.debian.org/python-team/modules/sphinx/-/commit/3c71964c17643ad0

> Now we need one further rule (which I think is already in place): Any
> sphinx extension should depend on python3-sphinx.

Any sphinx extension uses Sphinx Python API, so I think it is clear that
it should build-depend (and depend) on python3-sphinx.

> Together the case of sphinx extensions will work (i.e. packages will be
> cross-unsatisfiable).
>
> If you depend on sphinx and on a sphinx-extension, both will depend on
> python3-sphinx and that'll lock them to the same architecture.
> (python3-sphinx cannot be M-A:foreign. Only sphinx will be M-A:foreign.)
>
> > Piuparts is run as part of salsa-pipeline (see the green checkmark) :)
> >
> > I have just uploaded sphinx 2.4.3-3 which gets rid of alternatives and adds
> > Provides: sphinx (= ${binary:Version}).
>
> Cool.

So what are our next steps? I will make a Sphinx 3.x upload with updated
package description to experimental soon.

Do you want to prepare mass bug filing, or you prefer that we deal with the
packages in Python team (DPMT/PAPT) first?

Unfortunately I won't have much time in the following weeks, so if you want
some actions from me, I may be slow in replying.

--
Dmitry Shachnev


signature.asc
Description: PGP signature

Bug#961206: improve sphinx usage for cross building

[Dmitry Shachnev]

Hi Drew!

On Fri, May 29, 2020 at 01:34:16AM +0800, Drew Parsons wrote:
> Source: sphinx
> Followup-For: Bug #961206
>
> I've just updated numba from Build-Depends: python3-sphinx
> to Build-Depends: sphinx as recommended here (#961741).
>
> The change is giving me a lintian error:
>
> E: numba source: missing-build-dependency-for-dh-addon sphinxdoc => 
> python-sphinx | python3-sphinx | dh-sequence-sphinxdoc
>
> Should the lintian test be updated for the new sphinx structure (e.g.
> to include sphinx as an alternative)?

Good catch. Submitted a merge request for Lintian:

https://salsa.debian.org/lintian/lintian/-/merge_requests/315

--
Dmitry Shachnev


signature.asc
Description: PGP signature

Bug#961206: improve sphinx usage for cross building

[Drew Parsons]

Source: sphinx
Followup-For: Bug #961206

I've just updated numba from Build-Depends: python3-sphinx
to Build-Depends: sphinx as recommended here (#961741).

The change is giving me a lintian error:

E: numba source: missing-build-dependency-for-dh-addon sphinxdoc => 
python-sphinx | python3-sphinx | dh-sequence-sphinxdoc


Should the lintian test be updated for the new sphinx structure (e.g.
to include sphinx as an alternative)?

Bug#961206: improve sphinx usage for cross building

[Dmitry Shachnev]

On Wed, May 27, 2020 at 06:07:02PM +0200, Helmut Grohne wrote:
> > But the actual split is the last step (4th in your message). So at the first
> > step I just add Provides and don't split anything. Right?
>
> Yes, thank you for reminding me of what I wrote. I'm working on too many
> packages in parallel.
>
> Can we also document somewhere what dependency one should use? I'm not
> sure where one would do that though. Possibly a README? Sometimes this
> is done in the package description.

I have not documented it yet, but I can do it in the next upload.

Should I say something about the extensions? Maybe something like this:

| Build-depend on sphinx if your package uses /usr/bin/sphinx-* executables
| and does not use Sphinx extensions that depend on anything written in C
| or depend on specific Python interpreter architecture in any other way.
| Build-depend on python3-sphinx if that is not the case, or if your package
| uses Python API of sphinx.

Or I should leave only part about Python API, and don't mention extensions
because such packages will not cross-build anyway? Please suggest a better
wording if you can.

> How feasible would it be to ask lintian devs for help? Can we detect
> uses of sphinx-build in a reasonable manner to flag missing
> dependencies?

Maybe. That would be much more visible than the README.

> > Can you please quickly look at this commit and say if it makes sense to you?
> >
> > https://salsa.debian.org/python-team/modules/sphinx/-/commit/c51e0ad4908bb7d6
>
> I do not see and issues in that commit. I'd throw it at piuparts before
> daring to upload it though. piuparts should be able to catch the worst
> issues.

Piuparts is run as part of salsa-pipeline (see the green checkmark) :)

I have just uploaded sphinx 2.4.3-3 which gets rid of alternatives and adds
Provides: sphinx (= ${binary:Version}).

--
Dmitry Shachnev


signature.asc
Description: PGP signature

Bug#961206: improve sphinx usage for cross building

[Helmut Grohne]

Hi Dmitry,

On Wed, May 27, 2020 at 05:56:09PM +0300, Dmitry Shachnev wrote:
> If we can fix cross building in DPMT in an automated way, then why not do it?
> Of course it is not the first priority, we can do it after fixing all other
> packages.

I did mean to say that but got it wrong.

> But the actual split is the last step (4th in your message). So at the first
> step I just add Provides and don't split anything. Right?

Yes, thank you for reminding me of what I wrote. I'm working on too many
packages in parallel.

Can we also document somewhere what dependency one should use? I'm not
sure where one would do that though. Possibly a README? Sometimes this
is done in the package description.

How feasible would it be to ask lintian devs for help? Can we detect
uses of sphinx-build in a reasonable manner to flag missing
dependencies?

> Can you please quickly look at this commit and say if it makes sense to you?
> 
> https://salsa.debian.org/python-team/modules/sphinx/-/commit/c51e0ad4908bb7d6

I do not see and issues in that commit. I'd throw it at piuparts before
daring to upload it though. piuparts should be able to catch the worst
issues.

Helmut

Bug#961206: improve sphinx usage for cross building

[Dmitry Shachnev]

On Tue, May 26, 2020 at 07:42:30PM +0200, Helmut Grohne wrote:
> > It would be nice to have a better estimate of how many packages can be
> > fixed in an automated way in DPMT [1], how many packages cannot be fixed
> > at all (e.g. because they use sphinx from Python interface) and how many
> > packages are remaining.
>
> Given that many packages use python3 -m sphinx now, the breakage would
> be quite small actually. Using python3 -m sphinx would continue to just
> work after the split (though it would still break cross building). So I
> guess, we can simply remove DPMT from the calculation.

If we can fix cross building in DPMT in an automated way, then why not do it?
Of course it is not the first priority, we can do it after fixing all other
packages.

> > The first step (making python3-sphinx provide sphinx) is zero cost, so
> > I can do it quite soon.
>
> Doing so enables depending on sphinx, but python3-sphinx and sphinx
> actually need to be distinct packages, because sphinx should become
> Multi-Arch: foreign while python3-sphinx should not. You cannot express
> that using Provides.

But the actual split is the last step (4th in your message). So at the first
step I just add Provides and don't split anything. Right?

> > Do you know what is the process of switching from update-alternatives
> > to directly shipping the symlink? Can I just drop the postinst/postrm
> > scripts and add that symlink, or I need to somehow unregister the
> > alternative when the package is upgraded?
>
> I think you need to actually declare Conflicts (not just Breaks and
> Replaces) on any alternative (i.e. python-sphinx). Then, you'd remove
> the alternative in preinst (like you do in prerm already) and unpacking
> the symlinks should work.

Can you please quickly look at this commit and say if it makes sense to you?

https://salsa.debian.org/python-team/modules/sphinx/-/commit/c51e0ad4908bb7d6

--
Dmitry Shachnev


signature.asc
Description: PGP signature

Bug#961206: improve sphinx usage for cross building

[Helmut Grohne]

Hi Dmitry,

On Tue, May 26, 2020 at 02:46:28PM +0300, Dmitry Shachnev wrote:
> It would be nice to have a better estimate of how many packages can be
> fixed in an automated way in DPMT [1], how many packages cannot be fixed
> at all (e.g. because they use sphinx from Python interface) and how many
> packages are remaining.
> 
> [1] Ondřej Nový did the previous mass change that changed ‘sphinx-build’
> to ‘python3 -m sphinx’ in debian/rules, perhaps it would be easy for
> him to revert that change and at the same time update the build
> dependency.

Given that many packages use python3 -m sphinx now, the breakage would
be quite small actually. Using python3 -m sphinx would continue to just
work after the split (though it would still break cross building). So I
guess, we can simply remove DPMT from the calculation.

> The first step (making python3-sphinx provide sphinx) is zero cost, so
> I can do it quite soon.

Doing so enables depending on sphinx, but python3-sphinx and sphinx
actually need to be distinct packages, because sphinx should become
Multi-Arch: foreign while python3-sphinx should not. You cannot express
that using Provides.

> update-alternatives is no longer needed because Sphinx no longer supports
> Python 2.

I guessed that.

> Do you know what is the process of switching from update-alternatives
> to directly shipping the symlink? Can I just drop the postinst/postrm
> scripts and add that symlink, or I need to somehow unregister the
> alternative when the package is upgraded?

I think you need to actually declare Conflicts (not just Breaks and
Replaces) on any alternative (i.e. python-sphinx). Then, you'd remove
the alternative in preinst (like you do in prerm already) and unpacking
the symlinks should work.

Helmut

Bug#961206: improve sphinx usage for cross building

[Dmitry Shachnev]

Hi Simon!

On Thu, May 21, 2020 at 06:46:46PM +0100, Simon McVittie wrote:
> This would be reversing the recent recommendation to
> call `python3 -m sphinx` in preference to `sphinx-build` (e.g. in
>  and
> ). I don't
> fully understand what the reason for that recommendation was; does it
> still apply?

I think we switched to ‘python3 -m sphinx’ to make sure that the Python 3
version is always used, not the Python 2 one:

https://lists.debian.org/debian-python/2018/08/msg00076.html

As Sphinx no longer supports Python 3, I think this can be safely reverted.

--
Dmitry Shachnev


signature.asc
Description: PGP signature

Bug#961206: improve sphinx usage for cross building

[Dmitry Shachnev]

Hi Helmut!

On Thu, May 21, 2020 at 01:43:42PM +0200, Helmut Grohne wrote:
> Now the questions are:
>  * Is the requested sphinx (the cli) and python3-sphinx (the module)
>split a reasonable thing to do?
>  * Is the transition plan a reasonable thing to do?

I think it makes sense and is reasonable, yes.

And I definitely like the new plan more than the previously discussed
approach of making sphinx Architecture: any.

>  * Is this transition worth the cost (cross building vs changing lots of
>packages)?

It would be nice to have a better estimate of how many packages can be
fixed in an automated way in DPMT [1], how many packages cannot be fixed
at all (e.g. because they use sphinx from Python interface) and how many
packages are remaining.

[1] Ondřej Nový did the previous mass change that changed ‘sphinx-build’
to ‘python3 -m sphinx’ in debian/rules, perhaps it would be easy for
him to revert that change and at the same time update the build
dependency.

The first step (making python3-sphinx provide sphinx) is zero cost, so
I can do it quite soon.

>  * Can we get rid of the use of update-alternatives?

update-alternatives is no longer needed because Sphinx no longer supports
Python 2.

Do you know what is the process of switching from update-alternatives
to directly shipping the symlink? Can I just drop the postinst/postrm
scripts and add that symlink, or I need to somehow unregister the
alternative when the package is upgraded?

--
Dmitry Shachnev


signature.asc
Description: PGP signature

Bug#961206: improve sphinx usage for cross building

[Simon McVittie]

On Thu, 21 May 2020 at 13:43:42 +0200, Helmut Grohne wrote:
> A very rough
> sketch for this would be moving sphinx-build and the other tools to a
> new binary package maybe called simply "sphinx". Of course "sphinx"
> would depend on python3-sphinx, but given that sphinx would only cover
> the command line interface, sphinx could be marked Multi-Arch: foreign.
> Does this make sense in principle when one ignores the transition cost?

This would be reversing the recent recommendation to
call `python3 -m sphinx` in preference to `sphinx-build` (e.g. in
 and
). I don't
fully understand what the reason for that recommendation was; does it
still apply?

smcv

Bug#961206: improve sphinx usage for cross building

[Helmut Grohne]

Package: python3-sphinx
Version: 2.4.3-2
Severity: wishlist
User: debian-cr...@lists.debian.org
Usertags: cross-satisfiability

sphinx is becoming ever more popular and these days it is often used for
generating manual pages. That makes it required for building arch:any
packages (as opposed to arch:all *-doc) packages more often and this
affects cross building as a dependency on python3-sphinx is generally
not cross satisfiable. We cannot easily mark python3-sphinx Multi-Arch:
foreign either, so this leaves us in a difficult place.

This bug asks for improving the situation. It comes without a patch as I
seek consensus on how to make this work.

Generally, the rough idea would be marking python3-sphinx Multi-Arch:
foreign. Unfortunately, that is out of question, because the interface
of the python3-sphinx package inludes the sphinx Python module, which
happens to depend on markupsafe via jinja2. Since markupsafe is
architecture-dependent, there is no way to make python3-sphinx
Multi-Arch: foreign.

So the only option for improving the situation is making the interface
"smaller". In essence, this means providing the sphinx module and the
sphinx command line tools via different package names. A very rough
sketch for this would be moving sphinx-build and the other tools to a
new binary package maybe called simply "sphinx". Of course "sphinx"
would depend on python3-sphinx, but given that sphinx would only cover
the command line interface, sphinx could be marked Multi-Arch: foreign.
Does this make sense in principle when one ignores the transition cost?

Now getting there is not trivial. Simply moving sphinx-build an friends
to a new binary package is going to make a lot of packages FTBFS.
Obviously, that's not what we want. So the transition would roughly work
like this:
 1. python3-sphinx starts providing sphinx.
 2. We ask downstream packages to switch their python3-sphinx dependency
to sphinx iff they call it via sphinx-build.  This affects at most
1200 source packages. A lot of them are python modules and could be
quickly fixed in git by some DPMT member. The actual affected
packages could be determined with a partial archive rebuild.
 3. After a while and getting that number of 1200 down a little, we
could start a MBF at severity normal.
 4. Once the number of broken rdeps is down to around 100, we could do
the actual split and bump the remaining bugs to rc severity.

I doubt that this would finish in time for bullseye.

This does not solve cross building for advanced sphinx users where
sphinx extensions are involved. However that is not the majority of
sphinx users.

Since moving alternatives from one binary package to another is
difficult, I wonder whether we can rid of it now. Doing so would greatly
easy the projected steps.

Now the questions are:
 * Is the requested sphinx (the cli) and python3-sphinx (the module)
   split a reasonable thing to do?
 * Is the transition plan a reasonable thing to do?
 * Is this transition worth the cost (cross building vs changing lots of
   packages)?
 * Can we get rid of the use of update-alternatives?

Helmut