New submission from Brian Harring <ferri...@gmail.com>: Currently, shlex.shlex's internal read_token is a bit of a nasty mess to follow and pretty poorly performant.
The refactoring I'll be attaching essentially inverts the state machine approach read_token uses- instead, converting it over to a more procedural set of loops. The benefits of this is that it's far, far easier to track what the code is actually doing and the performance goes up fairly drastically via tightening the various for loops (for example, if doing quoting, you only care about escapes and the terminating quote- thus a loop there). End result, the refactored code actually has whitespace in it and still is shorter- specifically the tokenenizer was converted into a generator to preserve state internally, thus simplifying the code flow immensely. Beyond that, this should still be api compatible... and is 2x faster, passing tests. Note this patch is cut against 2.6.4; assuming folks are happy w/ the general approach, I'll rebase it to py3k trunk. >From there, I'd be willing to tackle the other shlex issues where applicable, but personally... not w/out the refactoring base, since that state machine is a pita to trace out. ---------- components: Library (Lib) files: shlex-py2.6.4-20091229.patch keywords: patch messages: 96987 nosy: ferringb severity: normal status: open title: shlex refactoring type: performance versions: Python 2.6, Python 2.7, Python 3.2 Added file: http://bugs.python.org/file15692/shlex-py2.6.4-20091229.patch _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue7594> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com