Bug#739300: [Python-modules-team] Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Yaroslav Halchenko]

severity 739300 normal
thanks


On Tue, 04 Mar 2014, Sebastian Ramacher wrote:
>  - lintian emits a privacy-breach-generic because of this [1].
> ...
> [1] http://lintian.debian.org/tags/privacy-breach-generic.html lists
> some packages that emit the warning for multiple possible MathJax
> URLs. I didn't check if they use sphinx to generate the documentation,
> but the list contains some python-*-doc which are the most likely
> candidates that could be using sphinx and sphinx.ext.mathjax.

I can only confirm that this is also an issue with python-dipy package
which now makes lintian spit out dozens of privacy-breach-uses-embedded-file
errors (not a warning).  IMHO it should be fixed within *sphinx*
ecosystem, and since it is a lintian error, not just a warning now, I
will raise the severity to at least normal for this issue.

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Ximin Luo]

Nicholas D Steeves:
> Hi!
> 
> On Sun, Dec 04, 2016 at 11:51:38AM +0300, Dmitry Shachnev wrote:
>> Hi Ximin!
>>
>> On Sat, Dec 03, 2016 at 01:49:00AM +, Ximin Luo wrote:
>>> Could dh_sphinxdoc not scan the html files for 

Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Nicholas D Steeves]

Hi!

On Sun, Dec 04, 2016 at 11:51:38AM +0300, Dmitry Shachnev wrote:
> Hi Ximin!
> 
> On Sat, Dec 03, 2016 at 01:49:00AM +, Ximin Luo wrote:
> > Could dh_sphinxdoc not scan the html files for 

Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Dmitry Shachnev]

Hi Ximin!

On Sat, Dec 03, 2016 at 01:49:00AM +, Ximin Luo wrote:
> Could dh_sphinxdoc not scan the html files for 

Bug#739300: [Python-modules-team] Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Ximin Luo]

Ximin Luo:
> On Mon, 11 Apr 2016 13:30:40 +0200 Dmitry Shachnev  wrote:
>> Hi Sandro,
>>
>> On Sat, Apr 09, 2016 at 01:27:03PM +0100, Sandro Tosi wrote:
>>> Is there any progress on this? I just got bitten by it:
>>>
>>> E: python-cycler-doc: privacy-breach-uses-embedded-file
>>> usr/share/doc/python-cycler-doc/html/_modules/cycler.html You may use
>>> libjs-mathjax package. (https://cdn.mathjax.org/mathjax/latest/mat
>>> hjax.js?config=tex-ams-mml_htmlormml)
>>>
>>> a quick grep on DPMT packages, shows already scipy and networkx has to
>>> manually symlink the file from libjs-mathjax to their local doc dir
>>>
>>> it would be extremely helpful if sphinx could do that for us.
>>
>> As I previously explained in this bug, you will have to replace the MathJax
>> URL in conf.py manually anyway — dh_sphinxdoc can't do that for you.
>>
>> Replacing it with path to packaged MathJax.js is usually enough. If it's not
>> (like when you want to serve the documentation via HTTP), then you can change
>> the URL to something in _static/, and you need *one* line to symlink that to
>> packaged MathJax. Because (in my opinion) this case is quite rare, because
>> the solution is just one line, and also because of potential problems with
>> directory symlinking, I also don't think this is something that could be done
>> in dh_sphinxdoc.
>>
> 
> Hi Dmitry, Gordon and I are trying to package ipywidgets [1] and are running 
> into this issue as well.
> 
> I don't understand your explanation above. [..]

Sorry, I just read through the whole thread, now I understand what you meant by 
"edit the conf".

However, I still think it would be better for dh_sphinxdoc to do it, so as to 
avoid every packager spending the time to do it myself. Yes it's just a few 
lines, but I (and each new packager) would have to spend about an hour figuring 
out what those lines are, and testing it to make sure it works. And these lines 
will multiply over many packages. Some people might do it slightly differently, 
then the code gets more and more confusing eventually.

So, I will try to write the patch anyhow.

BTW can someone point me to some discussion/documentation on precisely how/why 
symlinks to directories mess with dpkg? I was not aware of this - I have added 
quite a lot of directory-symlinks to some of my recent JS packages, and we will 
likely have to do this for ipywidgets, with mathjax...

> [..] None of the config files in ipywidgets contain a specific MathJax URL, 
> the remote URL we are getting is the default one from the sphinx.ext.mathjax 
> extension. So we have nothing to change in our own package..
> 
> Could dh_sphinxdoc not scan the html files for 

Bug#739300: [Python-modules-team] Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Ximin Luo]

On Mon, 11 Apr 2016 13:30:40 +0200 Dmitry Shachnev  wrote:
> Hi Sandro,
> 
> On Sat, Apr 09, 2016 at 01:27:03PM +0100, Sandro Tosi wrote:
> > Is there any progress on this? I just got bitten by it:
> >
> > E: python-cycler-doc: privacy-breach-uses-embedded-file
> > usr/share/doc/python-cycler-doc/html/_modules/cycler.html You may use
> > libjs-mathjax package. (https://cdn.mathjax.org/mathjax/latest/mat
> > hjax.js?config=tex-ams-mml_htmlormml)
> >
> > a quick grep on DPMT packages, shows already scipy and networkx has to
> > manually symlink the file from libjs-mathjax to their local doc dir
> >
> > it would be extremely helpful if sphinx could do that for us.
> 
> As I previously explained in this bug, you will have to replace the MathJax
> URL in conf.py manually anyway — dh_sphinxdoc can't do that for you.
> 
> Replacing it with path to packaged MathJax.js is usually enough. If it's not
> (like when you want to serve the documentation via HTTP), then you can change
> the URL to something in _static/, and you need *one* line to symlink that to
> packaged MathJax. Because (in my opinion) this case is quite rare, because
> the solution is just one line, and also because of potential problems with
> directory symlinking, I also don't think this is something that could be done
> in dh_sphinxdoc.
> 

Hi Dmitry, Gordon and I are trying to package ipywidgets [1] and are running 
into this issue as well.

I don't understand your explanation above. None of the config files in 
ipywidgets contain a specific MathJax URL, the remote URL we are getting is the 
default one from the sphinx.ext.mathjax extension. So we have nothing to change 
in our own package..

Could dh_sphinxdoc not scan the html files for 

Bug#739300: [Python-modules-team] Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Dmitry Shachnev]

Hi Sandro,

On Sat, Apr 09, 2016 at 01:27:03PM +0100, Sandro Tosi wrote:
> Is there any progress on this? I just got bitten by it:
>
> E: python-cycler-doc: privacy-breach-uses-embedded-file
> usr/share/doc/python-cycler-doc/html/_modules/cycler.html You may use
> libjs-mathjax package. (https://cdn.mathjax.org/mathjax/latest/mat
> hjax.js?config=tex-ams-mml_htmlormml)
>
> a quick grep on DPMT packages, shows already scipy and networkx has to
> manually symlink the file from libjs-mathjax to their local doc dir
>
> it would be extremely helpful if sphinx could do that for us.

As I previously explained in this bug, you will have to replace the MathJax
URL in conf.py manually anyway — dh_sphinxdoc can't do that for you.

Replacing it with path to packaged MathJax.js is usually enough. If it's not
(like when you want to serve the documentation via HTTP), then you can change
the URL to something in _static/, and you need *one* line to symlink that to
packaged MathJax. Because (in my opinion) this case is quite rare, because
the solution is just one line, and also because of potential problems with
directory symlinking, I also don't think this is something that could be done
in dh_sphinxdoc.

I also thought about changing Sphinx's own default to be the packaged URL.
However, we only want it for Debian packages only, not for stuff that i.e.
upstream developers build using our Sphinx package. But then there is no safe
way to determine if Sphinx is called from Debian package build (one may not
always be using dpkg-buildpackage or even debhelper, and instead calling
sphinx-build manually from build: target in debian/rules).

Better ideas are welcome, of course.

--
Dmitry Shachnev


signature.asc
Description: PGP signature

Bug#739300: [Python-modules-team] Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Sandro Tosi]

Is there any progress on this? I just got bitten by it:

E: python-cycler-doc: privacy-breach-uses-embedded-file
usr/share/doc/python-cycler-doc/html/_modules/cycler.html You may use
libjs-mathjax package. (https://cdn.mathjax.org/mathjax/latest/mat
hjax.js?config=tex-ams-mml_htmlormml)

a quick grep on DPMT packages, shows already scipy and networkx has to
manually symlink the file from libjs-mathjax to their local doc dir

it would be extremely helpful if sphinx could do that for us.

thanks!

On Mon, Feb 17, 2014 at 3:38 PM, Sebastian Ramacher
 wrote:
> Package: sphinx-common
> Version: 1.2.1+dfsg-3
> Severity: wishlist
>
> Documentation using sphinx.ext.mathjax defaults to an online copy of
> MathJax. It is possible to change mathjax_path in conf.py to use a
> packaged or local copy. I think it would be a nice addition to
> dh_sphinxdoc if it detected a local copy of MathJax.js and replaced it
> with symlinks to the packaged one.
>
> MathJax requires plenty of other files from libjs-mathjax, so simply
> symlinking MathJax.js was not enough in my tests and ended up linking
> /usr/share/javascript/mathjax into _static. I don't know if this is
> easily doable in dh_sphinxdoc, but since sphinx.ext.mathjax is a
> extension installed by python(3)-sphinx, I think dh_sphinxdoc is the
> appropriate tool to handle this.
>
> Regards
> --
> Sebastian Ramacher
>
> ___
> Python-modules-team mailing list
> python-modules-t...@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/python-modules-team



-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
G+: https://plus.google.com/u/0/+SandroTosi

Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Dmitry Shachnev]

Hi Sebastian,

On Tue, 4 Mar 2014 19:48:11 +0100, Sebastian Ramacher wrote:
 In breathe I replaced the MathJax URL with the full path of MathJax.js
 as provided by libjs-mathjax. It gets rid of the lintian warning and
 works if one views the HTML files directly in a local browser. After a
 discussion with Stuart Prescott in #d-python I learned that this is
 wrong: if the documentation is served via a web server, the tex formulas
 are not rendered anymore.

Thanks for explaining me why simply changing the URL won’t work.
But is that really a popular use-case?

 I guess I was hoping for a magical solution from dh_sphinxdoc as
 sphinx.ext.mathjax is provided by sphinx and already has some code in
 place to replace javascript libraries with the ones provided by other
 packages. So that all of this could be handled in one place instead of
 every package that uses sphinx.ext.mathjax.

But package maintainers will still need to patch the code to use the
local path. I think writing one line to debian/rules is easy enough
if you have already written the patch :)

Anyway, I think it is possible to implement this algorithm in
dh_sphinxdoc:

| If an HTML file in a package references _static/mathjax/MathJax.js in
| ‘script’ tag, and neither of _static/mathjax and _static/mathjax/MathJax.js
| is already a symlink, then replace the _static/mathjax directory with
| a symlink to /usr/share/javascript/mathjax/.

Will that work for you?

 [1] http://lintian.debian.org/tags/privacy-breach-generic.html lists
 some packages that emit the warning for multiple possible MathJax
 URLs. I didn't check if they use sphinx to generate the documentation,
 but the list contains some python-*-doc which are the most likely
 candidates that could be using sphinx and sphinx.ext.mathjax.

Thanks, I will file some bugs on these packages.

--
Dmitry Shachnev

signature.asc
Description: OpenPGP digital signature

Bug#739300: [Python-modules-team] Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Julian Taylor]

On 06.03.2014 09:07, Dmitry Shachnev wrote:
 
 Anyway, I think it is possible to implement this algorithm in
 dh_sphinxdoc:
 
 | If an HTML file in a package references _static/mathjax/MathJax.js in
 | ‘script’ tag, and neither of _static/mathjax and _static/mathjax/MathJax.js
 | is already a symlink, then replace the _static/mathjax directory with
 | a symlink to /usr/share/javascript/mathjax/.
 
 Will that work for you?
 
 

replacing a directory with a symlink is problematic as dpkg has special
rules for handling symlinks which can cause all kinds of upgrade issues.
The most common ones are files disappearing (see the plentora of missing
copyright bugs due to symlinking -doc packages) or unpack errors due to
ordering issues in the tarballs.

For this reason dh_linktree exists, please use that instead.
Another approach is to modify the path the webserver serves from, thats
what I do in IPython.


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#739300: [Python-modules-team] Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Dmitry Shachnev]

On Thu, 06 Mar 2014 19:00:09 +0100, Julian Taylor wrote:
 replacing a directory with a symlink is problematic as dpkg has special
 rules for handling symlinks which can cause all kinds of upgrade issues.
 The most common ones are files disappearing (see the plentora of missing
 copyright bugs due to symlinking -doc packages) or unpack errors due to
 ordering issues in the tarballs.

 For this reason dh_linktree exists, please use that instead.
 Another approach is to modify the path the webserver serves from, thats
 what I do in IPython.

I see two disadvantages in this approach:

- First, as Sebastian mentioned, “30k-ish symlinks is just too much for a
  documentation displaying one or two formulas”.

- Then, set of MathJax files changes with each release, so we will have
  to rebuild all such packages with every new MathJax release (otherwise
  some symlinks will be broken or missing).

The problem with replacing a directory with symlink can be easily solved
by adding a preinst script to the package.

--
Dmitry Shachnev

signature.asc
Description: OpenPGP digital signature

Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Sebastian Ramacher]

Hi Dmitry

On 2014-03-02 16:39:53, Dmitry Shachnev wrote:
 I have performed some queries on codesearch.debian.net and it seems that
 currently only two packages ship embedded copy of MathJax (mcrl2 and yelp):
 
 Your package breathe in NEW seems to be another one, and the only one using
 Sphinx. Given that you already fixes breathe, I think the impact of this
 bug is zero.

No, breathe doesn't ship an embedded copy of MathJax. My initial bug
report should have been better. Let me expand a bit:

 - breathe uses sphinx.ext.mathjax.
 - The final HTML files contain a script tag with src
   http://cdn.mathjax.org/mathjax/latest/MathJax.js per default.
 - lintian emits a privacy-breach-generic because of this [1].

In breathe I replaced the MathJax URL with the full path of MathJax.js
as provided by libjs-mathjax. It gets rid of the lintian warning and
works if one views the HTML files directly in a local browser. After a
discussion with Stuart Prescott in #d-python I learned that this is
wrong: if the documentation is served via a web server, the tex formulas
are not rendered anymore.

So I've tried to fix this in other ways. Simply creating a symlink for
MathJax.js in _static is unfortunately not enough as the other files
needed by MathJax won't be found. Using dh_linktree to create a symlink
tree instead I ended up with 30k-ish symlinks in the package. That's
just too much for a documentation displaying one or two formulas. If I'm
not missing another possible solution, this only leaves creating a
symlink to /usr/share/javascript/mathjax in _static [2].

I guess I was hoping for a magical solution from dh_sphinxdoc as
sphinx.ext.mathjax is provided by sphinx and already has some code in
place to replace javascript libraries with the ones provided by other
packages. So that all of this could be handled in one place instead of
every package that uses sphinx.ext.mathjax.

Cheers

[1] http://lintian.debian.org/tags/privacy-breach-generic.html lists
some packages that emit the warning for multiple possible MathJax
URLs. I didn't check if they use sphinx to generate the documentation,
but the list contains some python-*-doc which are the most likely
candidates that could be using sphinx and sphinx.ext.mathjax.

[2] Symlinking to directories always leaves a sour taste in my mouth.
Hello unhandled symlink to directory conversions and friends.
-- 
Sebastian Ramacher


signature.asc
Description: Digital signature

Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Dmitry Shachnev]

Hi Sebastian,

I have performed some queries on codesearch.debian.net and it seems that
currently only two packages ship embedded copy of MathJax (mcrl2 and yelp):

Your package breathe in NEW seems to be another one, and the only one using
Sphinx. Given that you already fixes breathe, I think the impact of this
bug is zero.

Also, the source of MathJax is quite large (especially if you ship PNG files),
so I would usually recommend people to remove that embedded copy from the
source tarball.

Anyway, I will keep this bug open and accept patches if someone decides to
write them. Some packages (i.e. calibre) already do that.

--
Dmitry Shachnev

signature.asc
Description: OpenPGP digital signature

Bug#739300: dh_sphinxdoc: Please replace MathJax.js with links to the packaged libjs-mathjax

[Dmitry Shachnev]

On Sun, 02 Mar 2014 16:39:53 +0400, Dmitry Shachnev wrote:
 Some packages (i.e. calibre) already do that.

That was related to previous paragraph (sorry), should be read as:

 I would usually recommend people to remove that embedded copy from the
 source tarball. Some packages (i.e. calibre) already do that.

--
Dmitry Shachnev

signature.asc
Description: OpenPGP digital signature