[issue6988] shlex.split() converts unicode input to UCS-4 output

Éric Araujo added the comment: $ ./python Python 2.7.2+ (2.7:27ae7d4e1983+, Oct 23 2011, 00:09:06) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import shlex >>> shlex.split(u'Hello, World!') ['Hello,', 'World!'] This was fixed indirectly by

[issue6988] shlex.split() converts unicode input to UCS-4 output

Terry J. Reedy added the comment: The discussion pretty much says this was a feature request, which is obsolete for 2.x. Not an issue for 3.x: >>> import shlex >>> shlex.split('Hello, World!' ) ['Hello,', 'World!'] -- nosy: +tjreedy resolution: -> out of date status: open -> closed ty

[issue6988] shlex.split() converts unicode input to UCS-4 output

Changes by Ezio Melotti : -- nosy: +ezio.melotti priority: -> normal ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscr

[issue6988] shlex.split() converts unicode input to UCS-4 output

Bill Fenner added the comment: Sorry, I didn't read the web documentation, only the module documentation, which doesn't mention Unicode. I'd agree that since it's a documented behavior, this bug can become: - an RFE for shlex to handle Unicode - meanwhile, if there will be any releases before

[issue6988] shlex.split() converts unicode input to UCS-4 output with varying byte order

Marc-Andre Lemburg added the comment: Amaury Forgeot d'Arc wrote: > > Amaury Forgeot d'Arc added the comment: > > (Presented this way, "my opinion" becomes difficult to stand... > OTOH the docs say that the module does not support Unicode, so it's not > strictly a bug) > http://docs.python.or

[issue6988] shlex.split() converts unicode input to UCS-4 output with varying byte order

Amaury Forgeot d'Arc added the comment: (Presented this way, "my opinion" becomes difficult to stand... OTOH the docs say that the module does not support Unicode, so it's not strictly a bug) http://docs.python.org/library/shlex.html Yes, shlex could be improved and encode unicode strings to as

[issue6988] shlex.split() converts unicode input to UCS-4 output with varying byte order

Antoine Pitrou added the comment: Hm, while the StringIO behaviour supposedly cannot be changed for backwards-compatibility reasons, we can probably improve shlex behaviour with unicode strings. -- nosy: +pitrou ___ Python tracker

[issue6988] shlex.split() converts unicode input to UCS-4 output with varying byte order

Bill Fenner added the comment: so, just to be clear, your position is that the output of shlex.split( u'Hello, World!' ) is *supposed* to be ['H\x00\x00\x00e\x00\x00\x00l\x00\x00\x00l\x00\x00\x00o\x00\x00\x00,\x00\x00\x00', '\x00\x00\x00W\x00\x00\x00o\x00\x00\x00r\x00\x00\x00l\x00\x00\x00d\x00\x

[issue6988] shlex.split() converts unicode input to UCS-4 output with varying byte order

Amaury Forgeot d'Arc added the comment: I'll take the opposite point of view: the bad behavior was introduced with 2.5.1 (issue1548891, r52302), and reverted for 2.5.2 because "it broke backwards compatibility with arbitrary read buffers" (issue1730114, r53831) The difference is in cStringIO:

[issue6988] shlex.split() converts unicode input to UCS-4 output with varying byte order

Bill Fenner added the comment: A colleague pointed out that the bad behavior was introduced in 2.5.2: Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import shlex >>> sh

[issue6988] shlex.split() converts unicode input to UCS-4 output with varying byte order

New submission from Bill Fenner : In python 2.5, shlex handled unicode input fine: Python 2.5.1 (r251:54863, Jun 15 2008, 18:24:51) [GCC 4.3.0 20080428 (Red Hat 4.3.0-8)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import shlex >>> shlex.split( u'Hello,