[issue23297] ‘tokenize.detect_encoding’ is confused between text and bytes: no ‘startswith’ method on a byte string
Pod added the comment: Not the OP, but I find this message a bug because it's confusing from the perspective of a user of the tokenize() function. If you give tokenize a readlines() that returns a str, you get this error message that confusingly states that something inside tokenize must be a string and NOT a bytes, even though the user gave readlines a string, not a bytes. It looks like an internal bug. Turns out it's because the contact changed from python2 to 3. Personally, I'd been accidentally reading the python2 page for the tokenize library instead of python3, and had been using tokenize.generate_tokens in my python 3 code which accepts a io.StringIO just fine. When I realising my mistake and switched to the python3 version of the page I noticed generate_tokens is no longer supported, even though the code I had was working, and I noticed that the definition of tokenize had changed to match the old generate_tokens (along with a subtle change in the definition of the acceptable readlines function). So when I switched from tokenize.generate_tokens to tokenize.tokenize to try and use the library as intended, I get the same error as OP. Perhaps OP made a similar mistake? To actually hit the error in question: $ cat -n temp.py 1 import tokenize 2 import io 3 4 5 byte_reader = io.BytesIO(btest bytes generate_tokens) 6 tokens = tokenize.generate_tokens(byte_reader.readline) 7 8 byte_reader = io.BytesIO(btest bytes tokenize) 9 tokens = tokenize.tokenize(byte_reader.readline) 10 11 byte_reader = io.StringIO(test string generate) 12 tokens = tokenize.generate_tokens(byte_reader.readline) 13 14 str_reader = io.StringIO(test string tokenize) 15 tokens = tokenize.tokenize(str_reader.readline) 16 17 $ python3 temp.py Traceback (most recent call last): File temp.py, line 15, in module tokens = tokenize.tokenize(str_reader.readline) File C:\work\env\python\Python34_64\Lib\tokenize.py, line 467, in tokenize encoding, consumed = detect_encoding(readline) File C:\work\env\python\Python34_64\Lib\tokenize.py, line 409, in detect_encoding if first.startswith(BOM_UTF8): TypeError: startswith first arg must be str or a tuple of str, not bytes -- nosy: +Pod ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23297 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Loop through a dict changing keys
On Sun, 16 Oct 2011 00:18:40 -0700, Jon Clements wrote:
On Oct 16, 12:53 am, PoD p...@internode.on.net wrote:
On Sat, 15 Oct 2011 11:00:17 -0700, Gnarlodious wrote:
What is the best way (Python 3) to loop through dict keys, examine
the string, change them if needed, and save the changes to the same
dict?
So for input like this:
{'Mobile': 'string', 'context': 'malicious code', 'order': '7',
'time': 'True'}
I want to booleanize 'True', turn '7' into an integer, escape
'malicious code', and ignore 'string'.
Any elegant Python way to do this?
-- Gnarlie
How about
data = {
'Mobile': 'string',
'context': 'malicious code',
'order': '7',
'time': 'True'}
types={'Mobile':str,'context':str,'order':int,'time':bool}
for k,v in data.items():
data[k] = types[k](v)
Bit of nit-picking, but:
bool('True')
True
bool('False')
True
bool('')
False
Oops :) Brain fade.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Loop through a dict changing keys
On Sat, 15 Oct 2011 11:00:17 -0700, Gnarlodious wrote:
What is the best way (Python 3) to loop through dict keys, examine the
string, change them if needed, and save the changes to the same dict?
So for input like this:
{'Mobile': 'string', 'context': 'malicious code', 'order': '7',
'time': 'True'}
I want to booleanize 'True', turn '7' into an integer, escape
'malicious code', and ignore 'string'.
Any elegant Python way to do this?
-- Gnarlie
How about
data = {
'Mobile': 'string',
'context': 'malicious code',
'order': '7',
'time': 'True'}
types={'Mobile':str,'context':str,'order':int,'time':bool}
for k,v in data.items():
data[k] = types[k](v)
--
http://mail.python.org/mailman/listinfo/python-list
Re: CAD file format specifications?
On Thu, 18 Jun 2009 13:39:28 +0200, Anthra Norell wrote: I had a look at Blender. It looks impressive too. It might be an alternative to Sketch Up. I'll worry about that later. My immediate need is a file conversion utility. A cursory inspection of Blender's menu tabs and the various help options didn't turn up a file-conversion utility. So, my question is: How do I convert a bunch of three-dimensional coordinates defining lines into a file format Sketch Up can read (skp, dwg, dxf, 3ds, ddf or dem)? Frederic If you look in the File/Import menu in Blender, you will see all the file types which it can load. The importing is done with python scripts, so if you are going to write a converter you might just as well write an import script for Blender. The raw import script shows how simple it can be. -- http://mail.python.org/mailman/listinfo/python-list
Re: Tabs are *MISUNDERSTOOD*, *EVIL* AND *STUPID*, end of discussion. (Re: Tabs versus Spaces in Source Code)
On Thu, 18 May 2006 08:30:03 +, Duncan Booth wrote: PoD wrote: How many levels of indentation does 12 spaces indicate? It could be 1,2,3,4,6 or 12. If you say it's 3 then you are _implying_ that each level is represented by 4 spaces. By reading the code I can see how many levels of indentation it represents. How many levels of indentation is 3 tabs? 3 levels in any code that you will find in the wild. No. That is precisely the problem: there is code in the wild which contains mixed space and tab indentation, and any time that happens 3 tabs could mean any number of indentations. I think it is universally accepted that mixed tabs and spaces is indeed **EVIL** I should have said any code using tabs exclusively. -- http://mail.python.org/mailman/listinfo/python-list
Re: Tabs are *MISUNDERSTOOD*, *EVIL* AND *STUPID*, end of discussion. (Re: Tabs versus Spaces in Source Code)
On Thu, 18 May 2006 10:33:58 +0200, Christophe wrote: PoD a écrit : On Wed, 17 May 2006 21:37:14 +0800, Andy Sy wrote: If tabs are easily misunderstood, then they are a MISfeature and they need to be removed. From the Zen of Python: Explicit is better than implicit... In the face of ambiguity, refuse the temptation to guess... Special cases aren't special enough to break the rules... Exactly. How many levels of indentation does 12 spaces indicate? It could be 1,2,3,4,6 or 12. If you say it's 3 then you are _implying_ that each level is represented by 4 spaces. Actually, who said you had to always use the same number of spaces to indent ? 12 = 6 + 6 = 4 + 4 + 4 but also 12 = 2 + 10 = 1 + 1 + 3 + 3 + 4 :D Thus supporting my assertion that space indenting is implicit not explicit. Spaces are evil. How many levels of indentation is 3 tabs? 3 levels in any code that you will find in the wild. No, it could be 3 levels or 3 tabs per level or 2 tabs for the first level and 1 tab for the second ... Could be but wouldn't be. Maybe what Python should do (but never will given the obsession with using spaces) is only allow one level of indentation increase per block so that def foo(): TABTABreturn 'bar' would return a syntax error -- http://mail.python.org/mailman/listinfo/python-list
Re: Tabs are *MISUNDERSTOOD*, *EVIL* AND *STUPID*, end of discussion. (Re: Tabs versus Spaces in Source Code)
On Fri, 19 May 2006 10:04:15 +0200, Christophe wrote: PoD a écrit : Maybe what Python should do (but never will given the obsession with using spaces) is only allow one level of indentation increase per block so that def foo(): TABTABreturn 'bar' would return a syntax error Which would make TAB mandatory for indentation. What about some freedom of choice ? Hey, if people are allowed to say that tabs should be banned, then I'm allowed to say they should be mandatory ;) -- http://mail.python.org/mailman/listinfo/python-list
Re: Tabs are *MISUNDERSTOOD*, *EVIL* AND *STUPID*, end of discussion. (Re: Tabs versus Spaces in Source Code)
On Wed, 17 May 2006 21:37:14 +0800, Andy Sy wrote: If tabs are easily misunderstood, then they are a MISfeature and they need to be removed. From the Zen of Python: Explicit is better than implicit... In the face of ambiguity, refuse the temptation to guess... Special cases aren't special enough to break the rules... Exactly. How many levels of indentation does 12 spaces indicate? It could be 1,2,3,4,6 or 12. If you say it's 3 then you are _implying_ that each level is represented by 4 spaces. How many levels of indentation is 3 tabs? 3 levels in any code that you will find in the wild. -- http://mail.python.org/mailman/listinfo/python-list
Re: getopt and options with multiple arguments
On Mon, 19 Dec 2005 02:29:41 -0800, [EMAIL PROTECTED] wrote: I want to be able to do something like: myscript.py * -o outputfile and then have the shell expand the * as usual, perhaps to hundreds of filenames. But as far as I can see, getopt can only get one argument with each option. In the above case, there isn't even an option string before the *, but even if there was, I don't know how to get getopt to give me all the expanded filenames in an option. Help! :) /David The convention is to put options first and then any other stuff such as filenames:- myscript.py -o outputfile * In this case getopt will return the options as one list and the rest as another list. -- http://mail.python.org/mailman/listinfo/python-list
Re: Recursive Property of Octal Numbers
On Fri, 30 Sep 2005 15:40:51 -0700, James Stroud wrote: I'm very curious about what is going on here. I'm sure my curiosity has something to do with ignorance of some fundamental concept of computer science (maybe that 8 is just a vertical infinity?): py b = '\xb6' py b[0] '\xb6' py b[0][0] '\xb6' py b[0][0][0] '\xb6' py b[0][0][0][0] '\xb6' py b[0][0][0][0][0] '\xb6' James b is a 1 character string. b[0] is a one character string which is the first character of b. Therefore b[0] == b. -- http://mail.python.org/mailman/listinfo/python-list
Re: function with variable arguments
On Fri, 13 May 2005 02:52:34 -0700, Xah Lee wrote: i wanted to define a function where the number of argument matters. Example: def Range(n): return range(n+1) def Range(n,m): return range(n,m+1) def Range(n,m,step): return range(n,m+1,step) this obvious doesn't work. The default argument like Range(n=1,m,step=1) obviously isn't a solution. can this be done in Python? or, must the args be changed to a list? def Range(n,m=None,step=1): if m is None: n,m = 0,n+1 else: n,m = n,m+1 return range(n,m,step) -- http://mail.python.org/mailman/listinfo/python-list
