Hello, in August 2012 I found a DoS vulnerability in expat and XML libraries in Python's standard library. Since then I have found several more issues. I have been working on fixes ever since.
The README of https://pypi.python.org/pypi/defusedxml contains detailed explanations of my research and all issues Blog post: http://blog.python.org/2013/02/announcing-defusedxml-fixes-for-xml.html Hotfixes: https://pypi.python.org/pypi/defusedxml https://pypi.python.org/pypi/defusedexpat Repositories: https://bitbucket.org/tiran/defusedxml https://bitbucket.org/tiran/defusedexpat https://bitbucket.org/tiran/expat CVE (work in progress) CVE-2013-1664 Unrestricted entity expansion induces DoS vulnerabilities in Python XML libraries (XML bomb) CVE-2013-1665 External entity expansion in Python XML libraries inflicts potential security flaws and DoS vulnerabilities Regards, Christian Extract from the documentation: Synopsis -------- The results of an attack on a vulnerable XML library can be fairly dramatic. With just a few hundred Bytes of XML data an attacker can occupy several Gigabytes of memory within seconds. An attacker can also keep CPUs busy for a long time with a small to medium size request. Under some circumstances it is even possible to access local files on your server, to circumvent a firewall, or to abuse services to rebound attacks to third parties. The attacks use and abuse less common features of XML and its parsers. The majority of developers are unacquainted with features such as processing instructions and entity expansions that XML inherited from SGML. At best they know about <!DOCTYPE> from experience with HTML but they are not aware that a document type definition (DTD) can generate an HTTP request or load a file from the file system. None of the issues is new. They have been known for a long time. Billion laughs was first reported in 2003. Nevertheless some XML libraries and applications are still vulnerable and even heavy users of XML are surprised by these features. It's hard to say whom to blame for the situation. It's too short sighted to shift all blame on XML parsers and XML libraries for using insecure default settings. After all they properly implement XML specifications. Application developers must not rely that a library is always configured for security and potential harmful data by default. Attack vectors ============== billion laughs / exponential entity expansion --------------------------------------------- The Billion Laughs attack -- also known as exponential entity expansion -- uses multiple levels of nested entities. The original example uses 9 levels of 10 expansions in each level to expand the string lol to a string of 3 * 10 9 bytes, hence the name "billion laughs". The resulting string occupies 3 GB (2.79 GiB) of memory; intermediate strings require additional memory. Because most parsers don't cache the intermediate step for every expansion it is repeated over and over again. It increases the CPU load even more. An XML document of just a few hundred bytes can disrupt all services on a machine within seconds. Example XML: <!DOCTYPE xmlbomb [ <!ENTITY a "1234567890" > <!ENTITY b "&a;&a;&a;&a;&a;&a;&a;&a;"> <!ENTITY c "&b;&b;&b;&b;&b;&b;&b;&b;"> <!ENTITY d "&c;&c;&c;&c;&c;&c;&c;&c;"> ]> <bomb>&d;</bomb> quadratic blowup entity expansion --------------------------------- A quadratic blowup attack is similar to a Billion Laughs attack; it abuses entity expansion, too. Instead of nested entities it repeats one large entity with a couple of ten thousand chars over and over again. The attack isn't as efficient as the exponential case but it avoids triggering countermeasures of parsers against heavily nested entities. Some parsers limit the depth and breadth of a single entity but not the total amount of expanded text throughout an entire XML document. A medium-sized XML document with a couple of hundred kilobytes can require a couple of hundred MB to several GB of memory. When the attack is combined with some level of nested expansion an attacker is able to achieve a higher ratio of success. <!DOCTYPE bomb [ <!ENTITY a "xxxxxxx... a couple of ten thousand chars"> ]> <bomb>&a;&a;&a;... repeat</bomb> external entity expansion (remote) ---------------------------------- Entity declarations can contain more than just text for replacement. They can also point to external resources by public identifiers or system identifiers. System identifiers are standard URIs. When the URI is a URL (e.g. a http:// locator) some parsers download the resource from the remote location and embed them into the XML document verbatim. Simple example of a parsed external entity: <!DOCTYPE external [ <!ENTITY ee SYSTEM "http://www.python.org/some.xml"> ]> <root>ⅇ</root> The case of parsed external entities works only for valid XML content. The XML standard also supports unparsed external entities with a NData declaration. External entity expansion opens the door to plenty of exploits. An attacker can abuse a vulnerable XML library and application to rebound and forward network requests with the IP address of the server. It highly depends on the parser and the application what kind of exploit is possible. For example: * An attacker can circumvent firewalls and gain access to restricted resources as all the requests are made from an internal and trustworthy IP address, not from the outside. * An attacker can abuse a service to attack, spy on or DoS your servers but also third party services. The attack is disguised with the IP address of the server and the attacker is able to utilize the high bandwidth of a big machine. * An attacker can exhaust additional resources on the machine, e.g. with requests to a service that doesn't respond or responds with very large files. * An attacker may gain knowledge, when, how often and from which IP address a XML document is accessed. An attacker could send mail from inside your network if the URL handler supports smtp:// URIs. external entity expansion (local file) -------------------------------------- External entities with references to local files are a sub-case of external entity expansion. It's listed as an extra attack because it deserves extra attention. Some XML libraries such as lxml disable network access by default but still allow entity expansion with local file access by default. Local files are either referenced with a file:// URL or by a file path (either relative or absolute). An attacker may be able to access and download all files that can be read by the application process. This may include critical configuration files, too. <!DOCTYPE external [ <!ENTITY ee SYSTEM "file:///PATH/TO/simple.xml"> ]> <root>ⅇ</root> DTD retrieval ------------- This case is similar to external entity expansion, too. Some XML libraries like Python's xml.dom.pulldown retrieve document type definitions from remote or local locations. Several attack scenarios from the external entity case apply to this issue as well. <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html> <head/> <body>text</body> </html> _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com