Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-24 Thread M.-A. Lemburg
Reminds me of the encoding attacks that were possible in earlier versions of Python... you could have e.g. an email processing script run the Python test suite by simply sending a specially crafted email :-) On 21.02.2013 13:04, Christian Heimes wrote: > Am 21.02.2013 11:32, schrieb Antoine Pitrou

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Eli Bendersky
On Thu, Feb 21, 2013 at 11:12 AM, Christian Heimes wrote: > Am 21.02.2013 19:39, schrieb Eli Bendersky: > > Just to clarify for my own curiosity. These attacks (e.g. > > http://en.wikipedia.org/wiki/Billion_laughs) have been known and public > > since 2003? > > Correct, see https://pypi.python.org

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Christian Heimes
Am 21.02.2013 19:39, schrieb Eli Bendersky: > Just to clarify for my own curiosity. These attacks (e.g. > http://en.wikipedia.org/wiki/Billion_laughs) have been known and public > since 2003? Correct, see https://pypi.python.org/pypi/defusedxml#synopsis third paragraph. All XML attacks in my analy

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Eli Bendersky
On Thu, Feb 21, 2013 at 9:23 AM, Stephen J. Turnbull wrote: > Jesse Noller writes: > > > I guess someone need to write a proof of concept exploit for you > > and release it into the wild. > > This is a bit ridiculous. This stuff looks easy enough that surely > Christian's post informed any mali

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Stephen J. Turnbull
Jesse Noller writes: > I guess someone need to write a proof of concept exploit for you > and release it into the wild. This is a bit ridiculous. This stuff looks easy enough that surely Christian's post informed any malicious body who didn't already know how to do it. If the exploit matters,

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Maciej Fijalkowski
On Thu, Feb 21, 2013 at 9:29 AM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 02/21/2013 01:53 AM, Antoine Pitrou wrote: >> On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano >> wrote: >>> >>> It's easy to forget that malware existed long before the Internet. >>> Th

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Maciej Fijalkowski
On Thu, Feb 21, 2013 at 6:35 AM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 02/20/2013 09:08 PM, Barry Warsaw wrote: >> On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote: >> >>> - make it possible to enable safer behaviour globally in at least >>> 2.7 and 3.3 (and p

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Antoine Pitrou
Le Thu, 21 Feb 2013 13:04:59 +0100, Christian Heimes a écrit : > Am 21.02.2013 11:32, schrieb Antoine Pitrou: > > You haven't proved that these were actual threats, nor how they > > actually worked. I'm gonna remain skeptical if there isn't anything > > more precise than "It highly depends on the

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Antoine Pitrou
Le Thu, 21 Feb 2013 13:19:54 +0100, Christian Heimes a écrit : > Am 21.02.2013 12:16, schrieb Antoine Pitrou: > > I don't know whether you are trying to be ironic but, for the > > record, proof of concepts needn't be "released into the wild" as > > long as they exist. > > Fun fact: > > In fact t

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Christian Heimes
Am 21.02.2013 12:16, schrieb Antoine Pitrou: > I don't know whether you are trying to be ironic but, for the record, > proof of concepts needn't be "released into the wild" as long as they > exist. Fun fact: In fact the abbreviation 'ap' doesn't stand for 'Antoine Pitrou' but for 'antipole'. I'm

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Christian Heimes
Am 21.02.2013 11:32, schrieb Antoine Pitrou: > You haven't proved that these were actual threats, nor how they > actually worked. I'm gonna remain skeptical if there isn't anything > more precise than "It highly depends on the parser and the application > what kind of exploit is possible". https:/

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Antoine Pitrou
Le Thu, 21 Feb 2013 06:05:52 -0500, Jesse Noller a écrit : > On Feb 21, 2013, at 5:32 AM, Antoine Pitrou > wrote: > > > Le Thu, 21 Feb 2013 11:18:35 +0100, > > Christian Heimes a écrit : > >> Am 21.02.2013 08:42, schrieb Antoine Pitrou: > >>> Sure, but in many instances, rebooting a machine is

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Jesse Noller
On Feb 21, 2013, at 5:32 AM, Antoine Pitrou wrote: > Le Thu, 21 Feb 2013 11:18:35 +0100, > Christian Heimes a écrit : >> Am 21.02.2013 08:42, schrieb Antoine Pitrou: >>> Sure, but in many instances, rebooting a machine is not >>> business-threatening. You will have a couple of minutes' downtim

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Antoine Pitrou
Le Thu, 21 Feb 2013 11:18:35 +0100, Christian Heimes a écrit : > Am 21.02.2013 08:42, schrieb Antoine Pitrou: > > Sure, but in many instances, rebooting a machine is not > > business-threatening. You will have a couple of minutes' downtime > > and that's all. Which is why the attack must be repeat

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Christian Heimes
Am 21.02.2013 08:42, schrieb Antoine Pitrou: > Sure, but in many instances, rebooting a machine is not > business-threatening. You will have a couple of minutes' downtime and > that's all. Which is why the attack must be repeated many times to be a > major annoyance. Is this business-threatening e

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Christian Heimes
Am 21.02.2013 10:23, schrieb Antoine Pitrou: > If you like being paranoid, there are other things than security to > be paranoid about: reference cycles, performance on micro-benchmarks, > memory consumption of docstrings, etc. :-) snappy(__doc__)? http://code.google.com/p/snappy/ Christian ___

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-21 Thread Antoine Pitrou
Le Thu, 21 Feb 2013 00:30:56 +0100, Christian Heimes a écrit : > Am 21.02.2013 00:08, schrieb Antoine Pitrou: > > Not everyone is a security nuts. > > But, but, but ... it's fun to be paranoid! You get so many new > potential enemies. :) If you like being paranoid, there are other things than se

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Antoine Pitrou
On Thu, 21 Feb 2013 02:29:08 -0500 Tres Seaver wrote: > > Antoine, > > A single, small,, malicious XML file can kill a machine (not just the > process parsing it) by sucking all available RAM. We are talking hard > lockup, reboot-to-fix-it sorts of DOC here. Sure, but in many instances, reboot

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/21/2013 01:53 AM, Antoine Pitrou wrote: > On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano > wrote: >> >> It's easy to forget that malware existed long before the Internet. >> The internet is just a transmission vector, it is not the source o

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Antoine Pitrou
On Thu, 21 Feb 2013 10:38:07 +1000 Nick Coghlan wrote: > On Thu, Feb 21, 2013 at 9:49 AM, Tres Seaver wrote: > > Two words: "hash randomization". If it applies to one, it applies to > > the other. > > Agreed. Christian's suggested approach sounds sane to me: > > - make it possible to enable s

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Antoine Pitrou
On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano wrote: > > It's easy to forget that malware existed long before the Internet. The > internet is just a transmission vector, it is not the source of malicious > files. The source of malicious files is *other people*, and unless you never > use

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Antoine Pitrou
On Wed, 20 Feb 2013 18:45:10 -0500 Donald Stufft wrote: > > No software you run on your computer grabs data from someone you don't trust > and it all validates that even though you trust them they haven't been > exploited? What the hell do you mean exactly? There are other reasons to validate d

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Stefan Behnel
Maciej Fijalkowski, 20.02.2013 21:17: > On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes wrote: >> Am 20.02.2013 17:25, schrieb Benjamin Peterson: >>> Are these going to become patches for Python, too? >> >> I'm working on it. The patches need to be discussed as they break >> backward compatibilit

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Barry Warsaw
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Feb 20, 2013, at 11:35 PM, Tres Seaver wrote: >I believe that the same rationale should apply as that for adding hash >randomization in 2.6.8: this is at least as bad a vulnerability, with >many more vectors of attack. Except that I really want

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/20/2013 09:08 PM, Barry Warsaw wrote: > On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote: > >> - make it possible to enable safer behaviour globally in at least >> 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) > > I want to

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Barry Warsaw
On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote: >- make it possible to enable safer behaviour globally in at least 2.7 >and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) I want to be fairly conservative with 2.6.9. -Barry ___ Python-Dev

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Fred Drake
On Wed, Feb 20, 2013 at 7:38 PM, Nick Coghlan wrote: > Christian's suggested approach sounds sane to me: Definitely. A strong +1 from me, FWIW these days. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein ___ Py

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Nick Coghlan
On Thu, Feb 21, 2013 at 9:49 AM, Tres Seaver wrote: > Two words: "hash randomization". If it applies to one, it applies to > the other. Agreed. Christian's suggested approach sounds sane to me: - make it possible to enable safer behaviour globally in at least 2.7 and 3.3 (and perhaps in 2.6 an

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Steven D'Aprano
On 21/02/13 10:22, Antoine Pitrou wrote: On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft wrote: On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, eve

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Donald Stufft
On Wednesday, February 20, 2013 at 6:22 PM, Antoine Pitrou wrote: > On Wed, 20 Feb 2013 18:21:22 -0500 > Donald Stufft mailto:donald.stu...@gmail.com)> > wrote: > > On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: > > > > It's not a distributed DoS issue, it's a severe DoS vulnera

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/20/2013 06:22 PM, Antoine Pitrou wrote: > On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft > wrote: >> On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilitie

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Jesse Noller
On Feb 20, 2013, at 6:22 PM, Antoine Pitrou wrote: > On Wed, 20 Feb 2013 18:21:22 -0500 > Donald Stufft wrote: >> On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Donald Stufft
On Wednesday, February 20, 2013 at 6:23 PM, Christian Heimes wrote: > We can add a function to the XML package tree that enables all restrictions: > > * limit expansion depths of nested entities > * limit total amount of expanded chars > * disable external entity expansion > * optionally force exp

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Antoine Pitrou
On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft wrote: > On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: > > > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A > > > single 1 kB XML document can kill virtually any machine, even servers > > > with more than

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Christian Heimes
Am 21.02.2013 00:08, schrieb Antoine Pitrou: > Not everyone is a security nuts. But, but, but ... it's fun to be paranoid! You get so many new potential enemies. :) Jerry Fletcher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Christian Heimes
Am 20.02.2013 23:56, schrieb Fred Drake: > While I'd hate to make XML processing more painful than it often is, there's > no injunction not to be reasonable. Security concerns and resource limits > are cross-cutting concerns, so it's not wrong to provide safe defaults. > > Doing so *will* be back

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Donald Stufft
On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: > > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A > > single 1 kB XML document can kill virtually any machine, even servers > > with more than hundred GB RAM. > > > > > Assuming an attacker can inject arbi

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Antoine Pitrou
On Wed, 20 Feb 2013 22:55:57 +0100 Christian Heimes wrote: > Am 20.02.2013 21:17, schrieb Maciej Fijalkowski: > > On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes > > wrote: > >> Am 20.02.2013 17:25, schrieb Benjamin Peterson: > >>> Are these going to become patches for Python, too? > >> > >> I

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Christian Heimes
Am 20.02.2013 23:45, schrieb R. David Murray: > I don't believe it does. The DTD URL is, if I remember correctly, > specified as an identifier. The fact that you can often also download the > DTD from the location specified by the identifier is a secondary effect. > > But, it's been a *long* tim

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Carl Meyer
On 02/20/2013 03:35 PM, Greg Ewing wrote: > Carl Meyer wrote: >> An XML parser that follows the XML standard is never safe to expose to >> untrusted input. > > Does the XML standard really mandate that a conforming parser > must blindly download any DTD URL given to it from the real > live interne

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Fred Drake
On Wed, Feb 20, 2013 at 5:45 PM, R. David Murray wrote: > (Wikipedia says: "Programs for reading documents may not be required to > read the external subset.", which would seem to confirm that.) Validating parsers are required to read the external subset; this doesn't apply to the parsers distrib

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread R. David Murray
On Thu, 21 Feb 2013 11:35:23 +1300, Greg Ewing wrote: > Carl Meyer wrote: > > An XML parser that follows the XML standard is never safe to expose to > > untrusted input. > > Does the XML standard really mandate that a conforming parser > must blindly download any DTD URL given to it from the rea

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Greg Ewing
Carl Meyer wrote: An XML parser that follows the XML standard is never safe to expose to untrusted input. Does the XML standard really mandate that a conforming parser must blindly download any DTD URL given to it from the real live internet? Somehow I doubt that. -- Greg _

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Christian Heimes
Am 20.02.2013 22:02, schrieb Carl Meyer: > Also, despite the title of this thread, the vulnerabilities include > fetching of external DTDs and entities (per standard), which opens up > attacks that are worse than just denial-of-service. In our initial > Django release advisory we carelessly lumped

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Christian Heimes
Am 20.02.2013 21:17, schrieb Maciej Fijalkowski: > On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes > wrote: >> Am 20.02.2013 17:25, schrieb Benjamin Peterson: >>> Are these going to become patches for Python, too? >> >> I'm working on it. The patches need to be discussed as they break >> backwa

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Carl Meyer
On 02/20/2013 01:53 PM, Skip Montanaro wrote: >> That's not very good. XML parsers are supposed to parse XML according >> to standards. Is the goal to have them actually do that, or just >> address DDOS issues? > > Having read through Christian's mail and several of his references, it > seems to m

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Skip Montanaro
> > I'm working on it. The patches need to be discussed as they break > > backward compatibility and AFAIK XML standards, too. > > That's not very good. XML parsers are supposed to parse XML according > to standards. Is the goal to have them actually do that, or just > address DDOS issues? Having

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Maciej Fijalkowski
On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes wrote: > Am 20.02.2013 17:25, schrieb Benjamin Peterson: >> Are these going to become patches for Python, too? > > I'm working on it. The patches need to be discussed as they break > backward compatibility and AFAIK XML standards, too. That's not

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Christian Heimes
Am 20.02.2013 17:25, schrieb Benjamin Peterson: > Are these going to become patches for Python, too? I'm working on it. The patches need to be discussed as they break backward compatibility and AFAIK XML standards, too. ___ Python-Dev mailing list Pyth

Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python

2013-02-20 Thread Benjamin Peterson
2013/2/19 Christian Heimes : > Hello, > > in August 2012 I found a DoS vulnerability in expat and XML libraries in > Python's standard library. Since then I have found several more issues. > I have been working on fixes ever since. > > The README of https://pypi.python.org/pypi/defusedxml contains