[issue1285086] urllib.quote is too slow

Serhiy Storchaka added the comment: Sorry, I perhaps missed your response, Senthil. Now committed and closed again. -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker

[issue1285086] urllib.quote is too slow

Roundup Robot added the comment: New changeset 4927899bea8d by Serhiy Storchaka in branch '2.7': Issue #1285086: Get rid of the refcounting hack and speed up urllib.unquote(). http://hg.python.org/cpython/rev/4927899bea8d New changeset 3cb07925fcb9 by Serhiy Storchaka in branch '3.2': Issue #128

[issue1285086] urllib.quote is too slow

Senthil Kumaran added the comment: Serhiy - Is there any technical issue that is holding up this patch? (I dont see any). If nothing is holding up and you are busy, I shall go ahead with committing this one. /cc flox -- ___ Python tracker

[issue1285086] urllib.quote is too slow

Changes by Ezio Melotti : -- nosy: +ezio.melotti ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.

[issue1285086] urllib.quote is too slow

Senthil Kumaran added the comment: > Serhiy Storchaka added the comment: >> >> + append = res.append >> >> And then use 'append'? > > This speed up unquote_to_bytes by 15%. Thanks for the response. In fact, writing the whole _hextobyte verbatim would further increase the speed, but that would be

[issue1285086] urllib.quote is too slow

Serhiy Storchaka added the comment: > Serhiy: One question. Is there a need to have - > > + append = res.append > > And then use 'append'? This speed up unquote_to_bytes by 15%. -- ___ Python tracker __

[issue1285086] urllib.quote is too slow

Senthil Kumaran added the comment: I wrongly minutes. Here is actual data of execution speeds. There is magnitude of difference (almost 130x faster). Measured on macbook pro with 2 cores and 4 Gig mem. Before Patch: $ ./python.exe -m timeit -s "import urllib.parse; x='a%20' * 10" "urllib.

Re: [issue1285086] urllib.quote is too slow

Applying this patch - I tried this hypothetical test. $ ./python.exe -m timeit -s "import urllib.parse; x='a%20' * 10" "urllib.parse.unquote(x)" 10 loops, best of 3: 205 msec per loop Without the patch, the above test will run for minutes! This creates a significant difference for extremely

[issue1285086] urllib.quote is too slow

Changes by Serhiy Storchaka : -- stage: needs patch -> patch review versions: +Python 3.3, Python 3.4 Added file: http://bugs.python.org/file29065/urllib_faster_unquote-2.7.patch ___ Python tracker __

[issue1285086] urllib.quote is too slow

Serhiy Storchaka added the comment: Proposed patches not only get rid of the refcount hack, but make unquote() and unquote_to_bytes() even significant faster for large strings. -- nosy: +serhiy.storchaka Added file: http://bugs.python.org/file29064/urllib_faster_unquote.patch _

[issue1285086] urllib.quote is too slow

Changes by Serhiy Storchaka : -- resolution: fixed -> stage: committed/rejected -> needs patch status: closed -> open ___ Python tracker ___ __

[issue1285086] urllib.quote is too slow

Maciej Fijalkowski added the comment: As per discussion on python-dev, this bug should probably be reopened and the patch maybe reverted as relying on the refcounting hack is both dodgy and hurts other implementations, like PyPy. -- nosy: +fijall __

[issue1285086] urllib.quote is too slow

Barry A. Warsaw added the comment: flox reverted in r83967 for 2.6.6. -- priority: release blocker -> status: open -> closed ___ Python tracker ___ __

[issue1285086] urllib.quote is too slow

Changes by Barry A. Warsaw : -- priority: low -> release blocker status: closed -> open ___ Python tracker ___ ___ Python-bugs-list

[issue1285086] urllib.quote is too slow

Barry A. Warsaw added the comment: Why was this merged to 2.6 after 2.6.6rc1 without approval? -- nosy: +barry ___ Python tracker ___ _

[issue1285086] urllib.quote is too slow

Florent Xicluna added the comment: Committed to 3.2 in r81271, after some additional tuning. Btw, I kept list comprehension in 3.2, because it is faster for small strings (even if the gain is ~10%). -- status: open -> closed ___ Python tracker

[issue1285086] urllib.quote is too slow

Changes by Senthil Kumaran : -- assignee: orsenthil -> flox resolution: -> fixed stage: patch review -> committed/rejected ___ Python tracker ___ _

[issue1285086] urllib.quote is too slow

Senthil Kumaran added the comment: On Mon, May 17, 2010 at 01:38:57PM +, Florent Xicluna wrote: > Committed in r81265 for 2.7. Thanks. That was interesting. Without resorting to any drastic changes like use of regex, interesting speed-up seems to have been achieved by using rsplit and map.

[issue1285086] urllib.quote is too slow

Changes by Florent Xicluna : Removed file: http://bugs.python.org/file17366/issue1285086_using_rstrip_v2.diff ___ Python tracker ___ ___ Pyt

[issue1285086] urllib.quote is too slow

Florent Xicluna added the comment: Committed in r81265 for 2.7. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubsc

[issue1285086] urllib.quote is too slow

Florent Xicluna added the comment: Proposed patches for quote and unquote on 2.7 and 3.2. -- Added file: http://bugs.python.org/file17367/issue1285086_using_rstrip_py3k.diff ___ Python tracker __

[issue1285086] urllib.quote is too slow

Changes by Florent Xicluna : Added file: http://bugs.python.org/file17366/issue1285086_using_rstrip_v2.diff ___ Python tracker ___ ___ Pytho

[issue1285086] urllib.quote is too slow

Florent Xicluna added the comment: Updated script for benchmarks (on 2.x and 3.x). Inspired by the "Tools/iobench" script. It benchmarks various quote/unquote implementations on 2.x and 3.x. On 2.7 the fastest implementation is something like: def quote(s): if not s or not s.rstri

[issue1285086] urllib.quote is too slow

Changes by Florent Xicluna : Removed file: http://bugs.python.org/file17299/urllib_quote_speed_test.py ___ Python tracker ___ ___ Python-bug

[issue1285086] urllib.quote is too slow

Changes by Florent Xicluna : Removed file: http://bugs.python.org/file17298/issue1285086_using_rstrip.diff ___ Python tracker ___ ___ Python

[issue1285086] urllib.quote is too slow

Changes by Florent Xicluna : Removed file: http://bugs.python.org/file17295/issue1285086_using_translate.diff ___ Python tracker ___ ___ Pyt

[issue1285086] urllib.quote is too slow

Changes by Florent Xicluna : Removed file: http://bugs.python.org/file17291/issue1285086_fast_quote.diff ___ Python tracker ___ ___ Python-b

[issue1285086] urllib.quote is too slow

Changes by Florent Xicluna : Added file: http://bugs.python.org/file17299/urllib_quote_speed_test.py ___ Python tracker ___ ___ Python-bugs-

[issue1285086] urllib.quote is too slow

Florent Xicluna added the comment: actually, there's a simpler implementation, using s.rstrip(always_safe + safe). It is as fast as the previous one. -- Added file: http://bugs.python.org/file17298/issue1285086_using_rstrip.diff ___ Python tracker

[issue1285086] urllib.quote is too slow

Changes by Florent Xicluna : Removed file: http://bugs.python.org/file17297/urllib_quote_speed_test.py ___ Python tracker ___ ___ Python-bug

[issue1285086] urllib.quote is too slow

Florent Xicluna added the comment: The speed test script did not work on 2.5 (because timeit.Timer does not accept a callable). Fixed version, which benchmarks the str.translate(...) version. Change the '_new_quote_setup' assignment to test other variants. -- Added file: http://bugs.p

[issue1285086] urllib.quote is too slow

Florent Xicluna added the comment: New patch, using str.translate instead of regexp. It is faster for normal cases (85% less time than stdlib quote), and the penalty for the real-quoting case is less than 5%. It should apply to 3.x with some adaptation. -- Added file: http://bugs.pyth

[issue1285086] urllib.quote is too slow

Senthil Kumaran added the comment: Lets also see how this fares in py3k (where quote function takes an encoding ) and possibly push it in. If there is any hesitation we can consult python-dev or wsgi groups where frameworks developers might review and voice concerns, if they have any. ---

[issue1285086] urllib.quote is too slow

Changes by Florent Xicluna : Added file: http://bugs.python.org/file17291/issue1285086_fast_quote.diff ___ Python tracker ___ ___ Python-bug

[issue1285086] urllib.quote is too slow

Changes by Florent Xicluna : Removed file: http://bugs.python.org/file17290/issue1285086_fast_quote.diff ___ Python tracker ___ ___ Python-b

[issue1285086] urllib.quote is too slow

Florent Xicluna added the comment: I've tested a variant of the previous patch. On my laptop, it gives good performance for simple cases, and the penalty for real-quoting case is very low. I've tested a short-circuit for the unquote() function too. -- nosy: +flox Added file: http://bu

[issue1285086] urllib.quote is too slow

Tres Seaver added the comment: I can only reiterate that Zope apps can call 'urllib.quote' dozens, hundreds, even thousands of times on a single request: the reason for the original bug report was that 'urllib.quote' was showing up frequently on profiling output for such requests. Short-circuit

[issue1285086] urllib.quote is too slow

Senthil Kumaran added the comment: I reviewed the patch and the speed test attached. Well, yes, the patch does achieve a certain level of speed improvement but I also saw that in cases when the quoting is really required (special characters, the current stdlib is faster). The speed improvem

[issue1285086] urllib.quote is too slow

Tres Seaver added the comment: Updated patch against the trunk attached. Note that I prefer unidiff myself, but was following a bit of guidance that Guido (used to, anyway) prefer context diffs. The updated speed test run against the stdlib with this patch shows only a 12% to 16% hit against

[issue1285086] urllib.quote is too slow

Antoine Pitrou added the comment: Tres, can you update your patch against SVN trunk? Also, please do an unified diff, we are mostly used to this format. -- stage: unit test needed -> patch review versions: +Python 2.7, Python 3.2 ___ Python tracker

[issue1285086] urllib.quote is too slow

Tres Seaver added the comment: I'm uploading a saner version of the speed test which uses timeit's support for passing a callable as a statement: it is much easier to see what the test is actually doing. On my machine, running against the release26-maint branch, my version runs anywhere from

[issue1285086] urllib.quote is too slow

Antoine Pitrou added the comment: If someone needs a faster quote(), it's probably easy to write an insanely fast C version... -- nosy: +pitrou ___ Python tracker ___

[issue1285086] urllib.quote is too slow

Changes by Daniel Diniz : -- nosy: +orsenthil stage: -> test needed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubs

[issue1285086] urllib.quote is too slow

Daniel Diniz <[EMAIL PROTECTED]> added the comment: This is what I found doing some timings: For the short-circuit path, the regexp can make quote 10x as fast in exceptional cases, even comparing to the faster version in trunk. The average win for short-circuit seems to be twice as fast as the m

[issue1285086] urllib.quote is too slow

Changes by Georg Brandl <[EMAIL PROTECTED]>: -- type: feature request -> performance _ Tracker <[EMAIL PROTECTED]> _ ___ Python-bugs-