Bug#287371: xsltproc: Probable memory leak (when using document()?)

[Diederik de Haas]

Control: tag -1 -security
Control: severity -1 wishlist

On vrijdag 17 februari 2023 23:47:55 CET you wrote:
> And the final message in this bug is that you run Out Of Memory, so the OOM
> killer does exactly what it needs to do: kill other programs.

It may be annoying, but the system is working as it should be/do, so I'm 
removing the 'security' tag.
If this bug turns out to be an actual security issue and there is thus a CVE, 
the tag can be added back.

> I'll leave changing the severity to the maintainer, but 'wishlist' seems
> appropriate to me.

It was earlier concluded, including by OP, that this was a wishlist type of 
bug and it was previously already set to that, so there is no need to postpone 
setting an appropriate severity level.

FTR: I'm not contesting that you may have found a valid bug.
In upstream's 1.1.36 there seems to be at least 1 fix for a potential memory 
leak (IIUC). 
But the lack of cooperation by OP combined with the severity and the age of 
the bug, did (apparently) bug me.

signature.asc
Description: This is a digitally signed message part.

Bug#287371: xsltproc: Probable memory leak (when using document()?)

[Diederik de Haas]

Control: tag -1 unreproducible

On Wed, 9 Feb 2005 17:12:21 +0100 Mike Hommey  wrote:
> On Dec 27, 2004, Vincent Lefevre  wrote:
> > Package: xsltproc
> > Version: 1.1.8-5
> > 
> > Here xsltproc takes up to 138 MB, making the whole system slow down
> > due to swapping. This problem occurs when generating my blog page,
> > where a document() is used for each blog item (this will change in
> > the future, but the current behavior shouldn't occur). The sources
> > are in a DocBook-based DTD that can be downloaded from
> > 
> >   http://www.vinc17.org/DTD/website.dtd
> > 
> > I'm not including the XML sources since this is quite complicated
> > (lots of inclusions and dependencies). But if the bug is not known,
> > I could try to build a simpler example.
> 
> How big is the document you load with document() ? How many times it
> gets loaded ? Could you provide me the files ?

The year is 2023, which means that 18 YEARS ago you were asked to provide a 
sample document ... which still hasn't happened.
It was reported again xslt version 1.1.8-5 with some mention of version 
2.6.16-1 of libxml2 ... both are ANCIENT, but no newer 'found' version was 
reported.

Just for fun, I downloaded your dtd (attached) and noticed a MathML entity 
which refers to DTD: "http://www.w3.org/TR/MathML2/dtd/mathml2.dtd
Trying to load that document gives a 404, but I found the following:
https://www.w3.org/TR/MathML2/appendixa.html#parsing.usingdtdt

My XML knowledge is a bit rusty, but that means your *custom made* DTD is 
invalid? 

On Sat, 19 Feb 2022 18:28:00 +0100 Vincent Lefevre  wrote:
> I'll test again (I've been using a fake DTD for the past 15 years).

IOW: you're not using your own DTD ... otherwise you might have noticed it's 
no longer valid?

What I do recall from when I was full into XML and related technologies is 
that it indeed uses a lot of memory.
As mentioned in 2005, 138MB is not considered *that* much.
And in 2023 that is even more true.

But you have a machine with *unspecified* but apparently very limited resources 
and there you try to load a *custom made* DTD which is probably quite complex 
(MathML can't be simple AFAIK).
"What could possibly go wrong (tm)"

On Sat, 19 Feb 2022 18:01:52 +0100 Mattia Rizzolo  wrote:
> If you believe so, and you confirmed that it hasn't been fixed in the
> past 15 years, could you please either (or both):
>  * report it to mitre's CVE form
>  * report it in https://gitlab.gnome.org/GNOME/libxml2/-/issues
> ?

That request was made a YEAR ago, but I'm not seeing a 'forwarded'? Or a link 
to a CVE item?

And the final message in this bug is that you run Out Of Memory, so the OOM 
killer does exactly what it needs to do: kill other programs.
But yet, you conclude that this is all xsltproc's fault?

There was no action on this bug for ~ 17 years, any requested information was 
not provided, the issue was not made reproducible, it was not reported 
upstream and I'm probably 'forgetting' a few items.

How in the world could this possibly be considered Release Critical?

I'll leave changing the severity to the maintainer, but 'wishlist' seems 
appropriate to me.

website.dtd
Description: application/xml-dtd


signature.asc
Description: This is a digitally signed message part.

Bug#287371: [xml/sgml-pkgs] Bug#287371: xsltproc: Probable memory leak (when using document()?)

[Vincent Lefevre]

On 2022-02-19 18:28:00 +0100, Vincent Lefevre wrote:
> I'll test again (I've been using a fake DTD for the past 15 years).

This has just happened. The consequence is that several unrelated
processes were killed by the OOM killer, including daemons!

[...]
Apr 25 02:44:53 zira systemd[6589]: dconf.service: A process of this unit has 
been killed by the OOM killer.
Apr 25 02:44:53 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSource/ldac_hq
Apr 25 02:44:53 zira systemd[1]: user@1000.service: A process of this unit has 
been killed by the OOM killer.
Apr 25 02:44:53 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSource/ldac_sq
Apr 25 02:44:53 zira systemd[6589]: pipewire.service: A process of this unit 
has been killed by the OOM killer.
Apr 25 02:44:53 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSource/ldac_mq
Apr 25 02:44:53 zira systemd[6589]: pipewire-media-session.service: A process 
of this unit has been killed by the OOM killer.
Apr 25 02:44:53 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSink/aptx_hd
Apr 25 02:44:53 zira systemd[6589]: pulseaudio.service: A process of this unit 
has been killed by the OOM killer.
Apr 25 02:44:53 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSource/aptx_hd
Apr 25 02:44:53 zira systemd[6589]: pipewire.service: Main process exited, 
code=killed, status=9/KILL
Apr 25 02:44:53 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSink/aptx
Apr 25 02:44:53 zira systemd[6589]: pipewire.service: Failed with result 
'oom-kill'.
Apr 25 02:44:53 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSource/aptx
Apr 25 02:44:55 zira systemd[1]: user@1000.service: A process of this unit has 
been killed by the OOM killer.
Apr 25 02:44:53 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSink/sbc
Apr 25 02:44:55 zira systemd[1]: session-204.scope: A process of this unit has 
been killed by the OOM killer.
Apr 25 02:44:53 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSource/sbc
Apr 25 02:44:55 zira systemd[6589]: Requested transaction contradicts existing 
jobs: Resource deadlock avoided
Apr 25 02:44:53 zira acpid[792]: input device has been disconnected, fd 23
Apr 25 02:44:55 zira systemd[6589]: gpg-agent.service: A process of this unit 
has been killed by the OOM killer.
Apr 25 02:44:55 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSink/sbc_xq_453
Apr 25 02:44:55 zira systemd[6589]: dbus.service: A process of this unit has 
been killed by the OOM killer.
Apr 25 02:44:55 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSource/sbc_xq_453
Apr 25 02:44:55 zira systemd[6589]: at-spi-dbus-bus.service: A process of this 
unit has been killed by the OOM killer.
Apr 25 02:44:55 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSink/sbc_xq_512
Apr 25 02:44:55 zira systemd[6589]: gvfs-daemon.service: A process of this unit 
has been killed by the OOM killer.
Apr 25 02:44:55 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSource/sbc_xq_512
Apr 25 02:44:55 zira systemd[6589]: pipewire-media-session.service: Main 
process exited, code=killed, status=9/KILL
Apr 25 02:44:55 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSink/sbc_xq_552
Apr 25 02:44:55 zira systemd[6589]: pipewire-media-session.service: Failed with 
result 'oom-kill'.
Apr 25 02:44:55 zira bluetoothd[241267]: Endpoint unregistered: sender=:1.100 
path=/MediaEndpoint/A2DPSource/sbc_xq_552
Apr 25 02:44:55 zira systemd[6589]: Stopped PipeWire Media Session Manager.
Apr 25 02:44:56 zira rtkit-daemon[826]: Successfully made thread 289323 of 
process 289323 owned by '1000' high priority at nice level -11.
Apr 25 02:44:55 zira systemd[6589]: pipewire-media-session.service: Consumed 
3.038s CPU time.
Apr 25 02:44:56 zira rtkit-daemon[826]: Supervising 1 threads of 1 processes of 
1 users.
Apr 25 02:44:55 zira systemd[6589]: dbus.service: Main process exited, 
code=killed, status=9/KILL
Apr 25 02:44:56 zira rtkit-daemon[826]: Supervising 1 threads of 1 processes of 
1 users.
Apr 25 02:44:55 zira systemd[6589]: dbus.service: Failed with result 'oom-kill'.
Apr 25 02:44:56 zira rtkit-daemon[826]: Supervising 1 threads of 1 processes of 
1 users.
Apr 25 02:44:55 zira systemd[6589]: gvfs-daemon.service: Main process exited, 
code=killed, status=9/KILL
Apr 25 02:44:56 zira rtkit-daemon[826]: Supervising 1 threads of 1 processes of 
1 users.
Apr 25 02:44:55 zira systemd[6589]: gvfs-daemon.service: Failed with result 
'oom-kill'.
Apr 25 02:44:56 zira rtkit-daemon[826]: Successfully made thread 289329 of 
process 

Bug#287371: [xml/sgml-pkgs] Bug#287371: xsltproc: Probable memory leak (when using document()?)

[Vincent Lefevre]

On 2022-02-19 18:01:52 +0100, Mattia Rizzolo wrote:
> On Thu, Feb 10, 2022 at 01:08:33PM +0100, Vincent Lefevre wrote:
> > This is no different than CVE-2013-0338 and CVE-2013-0339[*]. The
> > point is that from a small document, one can exhaust the memory
> > of the machine. CVE-2013-0338 and CVE-2013-0339 are about entity
> > expansion, but there are the same consequences with just loading
> > data in memory.
> > 
> > [*] https://www.openwall.com/lists/oss-security/2013/02/22/3
> 
> If you believe so, and you confirmed that it hasn't been fixed in the
> past 15 years, could you please either (or both):
>  * report it to mitre's CVE form
>  * report it in https://gitlab.gnome.org/GNOME/libxml2/-/issues
> ?

I'll test again (I've been using a fake DTD for the past 15 years).

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#287371: [xml/sgml-pkgs] Bug#287371: xsltproc: Probable memory leak (when using document()?)

[Mattia Rizzolo]

On Thu, Feb 10, 2022 at 01:08:33PM +0100, Vincent Lefevre wrote:
> This is no different than CVE-2013-0338 and CVE-2013-0339[*]. The
> point is that from a small document, one can exhaust the memory
> of the machine. CVE-2013-0338 and CVE-2013-0339 are about entity
> expansion, but there are the same consequences with just loading
> data in memory.
> 
> [*] https://www.openwall.com/lists/oss-security/2013/02/22/3

If you believe so, and you confirmed that it hasn't been fixed in the
past 15 years, could you please either (or both):
 * report it to mitre's CVE form
 * report it in https://gitlab.gnome.org/GNOME/libxml2/-/issues
?

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
More about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Bug#287371: xsltproc: Probable memory leak (when using document()?)

[Vincent Lefevre]

Control: severity -1 grave
Control: retitle -1 xsltproc: DTD should be cached when included several times, 
or used memory should be limited
Control: tags -1 security

On 2005-02-09 17:52:31 +0100, Mike Hommey wrote:
> On Wed, Feb 09, 2005 at 05:38:54PM +0100, Vincent Lefevre 
>  wrote:
> > On 2005-02-09 17:12:21 +0100, Mike Hommey wrote:
> > > How big is the document you load with document() ? How many times it
> > > gets loaded ? Could you provide me the files ?
> > 
> > The documents are small, but the DTD is very big (this is a DTD based
> > on DocBook + MathML). Currently, about 50 documents are included.
> > 
> > I wanted to post a followup, but hadn't had the time yet. FYI, I had
> > a discussion with Daniel on the LibXSLT mailing-list 10 days ago. In
> > short, for some reasons, the DTD structures are not reused each time
> > a new document is parsed. IMHO, this could be solved by some form of
> > cache (corresponding to the DTD + internal subset if any).
> > 
> > Technically, this bug could be regarded as a wishlist. But using so
> > much memory should be regarded as a bug IMHO, unless the other XSLT
> > processors have the same problem.
> > 
> > The title of the bug should be changed to something like "DTD
> > structures should be shared/cached in case of multiple inclusions"
> > (when possible, of course).
> 
> Thanks for the feedback.
> Note that such "optimization" bugs are not really *that* important, so i
> downgraded this bug to wishlist, even if a huge amount of memory is
> used. Also note that 138MB is not *that* much considering the number of
> documents and the DTD size.

This is no different than CVE-2013-0338 and CVE-2013-0339[*]. The
point is that from a small document, one can exhaust the memory
of the machine. CVE-2013-0338 and CVE-2013-0339 are about entity
expansion, but there are the same consequences with just loading
data in memory.

[*] https://www.openwall.com/lists/oss-security/2013/02/22/3

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#287371: xsltproc: Probable memory leak (when using document()?)

[Mike Hommey]

On Mon, Dec 27, 2004 at 01:11:49PM +0100, Vincent Lefevre [EMAIL PROTECTED] 
wrote:
 Package: xsltproc
 Version: 1.1.8-5
 Severity: important
 
 Here xsltproc takes up to 138 MB, making the whole system slow down
 due to swapping. This problem occurs when generating my blog page,
 where a document() is used for each blog item (this will change in
 the future, but the current behavior shouldn't occur). The sources
 are in a DocBook-based DTD that can be downloaded from
 
   http://www.vinc17.org/DTD/website.dtd
 
 I'm not including the XML sources since this is quite complicated
 (lots of inclusions and dependencies). But if the bug is not known,
 I could try to build a simpler example.

How big is the document you load with document() ? How many times it
gets loaded ? Could you provide me the files ?

Mike


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Bug#287371: xsltproc: Probable memory leak (when using document()?)

[Mike Hommey]

retitle 287371 DTD should be cached when included several times
severity 287371 wishlist
tag 287371 upstream
thanks

On Wed, Feb 09, 2005 at 05:38:54PM +0100, Vincent Lefevre [EMAIL PROTECTED] 
wrote:
 On 2005-02-09 17:12:21 +0100, Mike Hommey wrote:
  How big is the document you load with document() ? How many times it
  gets loaded ? Could you provide me the files ?
 
 The documents are small, but the DTD is very big (this is a DTD based
 on DocBook + MathML). Currently, about 50 documents are included.
 
 I wanted to post a followup, but hadn't had the time yet. FYI, I had
 a discussion with Daniel on the LibXSLT mailing-list 10 days ago. In
 short, for some reasons, the DTD structures are not reused each time
 a new document is parsed. IMHO, this could be solved by some form of
 cache (corresponding to the DTD + internal subset if any).
 
 Technically, this bug could be regarded as a wishlist. But using so
 much memory should be regarded as a bug IMHO, unless the other XSLT
 processors have the same problem.
 
 The title of the bug should be changed to something like DTD
 structures should be shared/cached in case of multiple inclusions
 (when possible, of course).

Thanks for the feedback.
Note that such optimization bugs are not really *that* important, so i
downgraded this bug to wishlist, even if a huge amount of memory is
used. Also note that 138MB is not *that* much considering the number of
documents and the DTD size.

Mike


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Bug#287371: xsltproc: Probable memory leak (when using document()?)

[Vincent Lefevre]

On 2005-02-09 17:52:31 +0100, Mike Hommey wrote:
 retitle 287371 DTD should be cached when included several times

To be more accurate: this is the internal structure related to the
DTD (and internal subset) that should be cached (to be reused when
the DTD with internal subset is the same, thus not taking additional
memory when a second document is processed).

 Note that such optimization bugs are not really *that* important,

Well, it is important on machines that don't have enough memory.

 so i downgraded this bug to wishlist, even if a huge amount of
 memory is used. Also note that 138MB is not *that* much considering
 the number of documents and the DTD size.

By caching the DTD structures, one could gain something like a
factor 1000 on the asymptotic memory usage with small documents
(3 KB vs 3 MB for the DTD itself). This is quite significant.

-- 
Vincent Lefèvre [EMAIL PROTECTED] - Web: http://www.vinc17.org/
100% accessible validated (X)HTML - Blog: http://www.vinc17.org/blog/
Work: CR INRIA - computer arithmetic / SPACES project at LORIA


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Bug#287371: xsltproc: Probable memory leak (when using document()?)

[Vincent Lefevre]

On 2005-02-10 01:29:38 +0100, Mike Hommey wrote:
 Machines that don't have enough memory can't run OpenOffice.Org. Will
 you file an important bug there as well ?

No, because OpenOffice.Org doesn't waste memory (it's quite memory
hungry, but this is expected, as it's a complex software). With
xsltproc, if one considers the sum of the sizes of all source data,
the required memory for the processing may be something like 1000
times larger, without any theoretical reason.

-- 
Vincent Lefèvre [EMAIL PROTECTED] - Web: http://www.vinc17.org/
100% accessible validated (X)HTML - Blog: http://www.vinc17.org/blog/
Work: CR INRIA - computer arithmetic / SPACES project at LORIA


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Bug#287371: [xml/sgml-pkgs] Bug#287371: xsltproc: Probable memory leak (when using document()?)

[Vincent Lefevre]

On 2004-12-31 14:15:42 +0900, Mike Hommey wrote:
 On Fri, Dec 31, 2004 at 02:40:54AM +0100, Vincent Lefevre [EMAIL PROTECTED] 
 wrote:
  On 2004-12-30 14:05:06 +0900, Mike Hommey wrote:
   Can you try with xsltproc from the experimental distribution? I know
   several memleaks have been fixed there and in libxml2.
  
  Unfortunately, there's no package for PowerPC yet.
 
 Can't you try to build it ?

I could try on an x86 machine where I've installed the experimental
libxml2 package (version 2.6.16-1). The problem is still there.

-- 
Vincent Lefèvre [EMAIL PROTECTED] - Web: http://www.vinc17.org/
100% accessible validated (X)HTML - Blog: http://www.vinc17.org/blog/
Work: CR INRIA - computer arithmetic / SPACES project at LORIA


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]