[issue26843] tokenize does not include Other_ID_Start or Other_ID_Continue in identifier

2016-04-25 Thread Joshua Landau
Joshua Landau added the comment: Sorry, I'd stumbled on my old comment on the closed issue and completely forgot about the *last* time I did the same thing. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/i

[issue26843] tokenize does not include Other_ID_Start or Other_ID_Continue in identifier

2016-04-24 Thread Joshua Landau
New submission from Joshua Landau: This is effectively a continuation of https://bugs.python.org/issue9712. The line in Lib/tokenize.py Name = r'\w+' must be changed to a regular expression that accepts Other_ID_Start at the start and Other_ID_Continue elsewhere. Hence tokenize does

[issue21593] Clarify re.search documentation first match

2015-10-25 Thread Joshua Landau
Changes by Joshua Landau <joshua.landau...@gmail.com>: -- versions: +Python 3.6 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python

[issue21593] Clarify re.search documentation first match

2015-05-14 Thread Joshua Landau
Joshua Landau added the comment: This should also be applied to regex.search's docstring. https://docs.python.org/3.5/library/re.html#re.regex.search -- resolution: fixed - status: closed - open ___ Python tracker rep...@bugs.python.org http

[issue24194] tokenize yield an ERRORTOKEN if an identifier uses Other_ID_Start or Other_ID_Continue

2015-05-14 Thread Joshua Landau
New submission from Joshua Landau: This is valid: ℘· = 1 print(℘·) # 1 But this gives an error token: from io import BytesIO from tokenize import tokenize stream = BytesIO(℘·.encode(utf-8)) print(*tokenize(stream.read), sep=\n) # TokenInfo(type=56 (ENCODING

[issue2292] Missing *-unpacking generalizations

2015-05-07 Thread Joshua Landau
Joshua Landau added the comment: There is a change as part of this to make dict building more like list and set building, which both have this behaviour. The same changes have likely occurred before whenever BUILD_LIST and BUILD_SET were introduced, and this behaviour seems particularly

[issue9712] tokenize yield an ERRORTOKEN if the identifier starts with a non-ascii char

2015-04-12 Thread Joshua Landau
Joshua Landau added the comment: This doesn't seem to be a complete fix; the regex used does not include Other_ID_Start or Other_ID_Continue from https://docs.python.org/3.5/reference/lexical_analysis.html#identifiers Hence tokenize does not accept '℘·'. Credit to modchan from http

[issue2292] Missing *-unpacking generalizations

2015-02-02 Thread Joshua Landau
Joshua Landau added the comment: I don't know the etiquette rules for the issue tracker, but I'd really appreciate having something to debug -- it's working for me, you see. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org

[issue2292] Missing *-unpacking generalizations

2015-01-30 Thread Joshua Landau
Joshua Landau added the comment: Special-cased `(*i for i in x)` to use YIELD_FROM instead of looping. Speed improved, albeit still only half as fast as chain.from_iterable. Fixed error message check in test_syntax and removed semicolons. -- Added file: http://bugs.python.org

[issue2292] Missing *-unpacking generalizations

2015-01-26 Thread Joshua Landau
Joshua Landau added the comment: Quick-fix for Guido's bug attached. I'm not familiar with this part of the code, yet, so take this tentatively. I just changed while (containers 1) { to while (containers) { --- @Guido My comments were assuming `f(**x for x in y)` meant `f({**x

[issue2292] Missing *-unpacking generalizations

2015-01-26 Thread Joshua Landau
Joshua Landau added the comment: Update for the error messages fix. I've put aside the idea of unifying things for now because there are a couple of interdependencies I wasn't expecting and I absolutely don't want the fast-path for f(x) to get slower. -- Added file: http

[issue2292] Missing *-unpacking generalizations

2015-01-26 Thread Joshua Landau
Joshua Landau added the comment: If we're supporting f(**x for x in y) surely we should also support f(x: y for x, y in z) I personally don't like this idea. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2292

[issue23316] Incorrect evaluation order of function arguments with *args

2015-01-25 Thread Joshua Landau
New submission from Joshua Landau: It is claimed that all expressions are evaluated left-to-right, including in functions¹. However, f(*a(), b=b()) will evaluate b() before a(). ¹ https://docs.python.org/3/reference/expressions.html#evaluation-order -- components: Interpreter

[issue2292] Missing *-unpacking generalizations

2015-01-25 Thread Joshua Landau
Joshua Landau added the comment: Amazing, thanks. I also just uncovered http://bugs.python.org/issue23316; we'll need to support a patch for that. In fact, bad evaluation order is why I haven't yet gotten down my unification strategy. I wouldn't worry about extra opcodes when using *args

[issue2292] Missing *-unpacking generalizations

2015-01-22 Thread Joshua Landau
Joshua Landau added the comment: I've looked at BUILD_MAP(n). It seems to work and has speed improvements but: - I was wrong about the 16-bit int thing. It turns out CPython is happily treating them as 32 bit as long as they are prefixed by an EXTENDED_ARG bytecode https://docs.python.org

[issue2292] Missing *-unpacking generalizations

2015-01-22 Thread Joshua Landau
Joshua Landau added the comment: Why would that simplify things? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2292 ___ ___ Python-bugs-list

[issue2292] Missing *-unpacking generalizations

2015-01-22 Thread Joshua Landau
Joshua Landau added the comment: I phrased that badly. Whilst I can see minor simplifications to BUILD_MAP_UNPACK, the only way to add more information to CALL_FUNCTION_XXX would be through EXTENDED_ARG. This seems like it would outweigh any benefits, and the tiny duplication of error

[issue2292] Missing *-unpacking generalizations

2015-01-22 Thread Joshua Landau
Joshua Landau added the comment: We wouldn't actually need to raise it from somewhere else; the line numbering and frame are already correct. The only difficulty is that the traceback currently says # func(a=1, **{'a': 1}) TypeError: func() got multiple values for keyword argument

[issue2292] Missing *-unpacking generalizations

2015-01-22 Thread Joshua Landau
Joshua Landau added the comment: The function object that's on the stack. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2292 ___ ___ Python

[issue2292] Missing *-unpacking generalizations

2015-01-22 Thread Joshua Landau
Joshua Landau added the comment: Just before any arguments to the function. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2292 ___ ___ Python

[issue2292] Missing *-unpacking generalizations

2015-01-22 Thread Joshua Landau
Joshua Landau added the comment: No, that happens in CALL_FUNCTION_KW: import dis dis.dis(f(x=1, **{'x': 1})) 1 0 LOAD_NAME0 (f) 3 LOAD_CONST 0 ('x') 6 LOAD_CONST 1 (1) 9 LOAD_CONST

[issue2292] Missing *-unpacking generalizations

2015-01-22 Thread Joshua Landau
Joshua Landau added the comment: I imagine it like (in the map unpacking code) func_offset = (oparg 8) 0xFF; num_maps = oparg 0xFF; // later if (func_offset) { // do checks if (repeated_argument) { raise_error_from_function(PEEK(func_offset

[issue2292] Missing *-unpacking generalizations

2015-01-22 Thread Joshua Landau
Joshua Landau added the comment: The stack will have the function, then any number of positional arguments, then optionally an *args, then any number (= 2) of maps to unpack. To get to the function, you need to know the sum count of all

[issue2292] Missing *-unpacking generalizations

2015-01-21 Thread Joshua Landau
Joshua Landau added the comment: I think I've fixed the memory leaks (plural). There were also a host of other problems with the _UNPACK opcodes in ceval. Here are the things I remember fixing, although I think I did slightly more: - Not throwing an error when PyDict_New or PyDict_Update

[issue2292] Missing *-unpacking generalizations

2015-01-21 Thread Joshua Landau
Joshua Landau added the comment: The _UNPACK opcodes are new in this changelist. Yup, but they're used in the other unpacking syntax too: (*(1, 2, 3), *(4, 5, 6)) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2292

[issue2292] Missing *-unpacking generalizations

2015-01-21 Thread Joshua Landau
Joshua Landau added the comment: According to the standard, int can be only 16 bits long so that only leaves 255/255. However, if the offset is on top of the dictionary count, this is easily enough to clear the limits for the maximum function size (worst case is a merge of 255 dicts

[issue2292] Missing *-unpacking generalizations

2015-01-21 Thread Joshua Landau
Joshua Landau added the comment: Functions are already limited to 255 arguments, so I don't think so. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2292

[issue2292] Missing *-unpacking generalizations

2015-01-21 Thread Joshua Landau
Joshua Landau added the comment: We wouldn't want to replace STORE_MAP since that's used in dictionary comprehensions, but replacing BUILD_MAP with BUILD_MAP(n) sounds like a great idea. -- ___ Python tracker rep...@bugs.python.org http

[issue2292] Missing *-unpacking generalizations

2015-01-21 Thread Joshua Landau
Joshua Landau added the comment: Good catch. CALL_FUNCTION seems to split its opcode into two to give it a positional-keyword pair so this seems fine. I'd hope we can do the same thing; personally I would do: BUILD_MAP_UNPACK( position_of_function_in_stack_or_0 8

[issue2292] Missing *-unpacking generalizations

2015-01-21 Thread Joshua Landau
Joshua Landau added the comment: Some of the tests seemed to be failing simply because they were incorrect. This fixes that. -- Added file: http://bugs.python.org/file37806/starunpack12.diff ___ Python tracker rep...@bugs.python.org http

[issue2292] Missing *-unpacking generalizations

2015-01-20 Thread Joshua Landau
Joshua Landau added the comment: The problem seems to be that with the removal of -else if (TYPE(ch) == STAR) { -vararg = ast_for_expr(c, CHILD(n, i+1)); -if (!vararg) -return NULL; -i++; -} -else if (TYPE(ch

[issue2292] Missing *-unpacking generalizations

2015-01-20 Thread Joshua Landau
Joshua Landau added the comment: This was a rather minor fix; I basically moved from STORE_SUBSCR to STORE_MAP and fixed a BUILD_MAP opcode. -- Added file: http://bugs.python.org/file37795/starunpack7.diff ___ Python tracker rep...@bugs.python.org

[issue2292] Missing *-unpacking generalizations

2015-01-20 Thread Joshua Landau
Joshua Landau added the comment: Aye, I'd done so (see starunpack7.diff). It was the fuss to reapply it ontop of your newer diff and making sure I'd read at least *some* of the devguide before barging on. Anyhow, here's another small fix to deal with the [*[0] for i in [0]] problem. I'm

[issue2292] Missing *-unpacking generalizations

2015-01-20 Thread Joshua Landau
Joshua Landau added the comment: I'm getting f(x=5, **{'x': 1}, **{'x': 3}, y=2) Traceback (most recent call last): File stdin, line 1, in module TypeError: f() got multiple values for keyword argument 'x' Which, as I understand, is the correct result. I'm using starunpack8

[issue2292] Missing *-unpacking generalizations

2015-01-20 Thread Joshua Landau
Joshua Landau added the comment: The problem with using STORE_MAP is you create a new dict for each keyword argument in that situation. You don't; if you look at the disassembly for producing a built-in dict (dis.dis('{1:2, 2:3, 3:4}')) you'll see they use STORE_MAP too. STORE_MAP seems

[issue2292] Missing *-unpacking generalizations

2015-01-20 Thread Joshua Landau
Joshua Landau added the comment: I think I've got it working; I'm just working out how to make a patch and adding a test or two. I think I'll also need to sign the contributor agreement. While I'm at it, here are a few other deviations from the PEP: - {*()} and {**{}} aren't supported

[issue2292] Missing *-unpacking generalizations

2015-01-20 Thread Joshua Landau
Joshua Landau added the comment: 2 here as well: 15 LOAD_CONST 2 ('w') 18 LOAD_CONST 3 (1) 21 BUILD_MAP1 24 LOAD_CONST 4 (2) 27 LOAD_CONST 5 ('x') 30 STORE_MAP 31 BUILD_MAP1 34 LOAD_CONST

[issue2292] Missing *-unpacking generalizations

2015-01-20 Thread Joshua Landau
Joshua Landau added the comment: I take it back; that just causes f(**{}, c=2) XXX lineno: 1, opcode: 105 Traceback (most recent call last): File stdin, line 1, in module SystemError: unknown opcode -- ___ Python tracker rep

[issue2292] Missing *-unpacking generalizations

2015-01-20 Thread Joshua Landau
Joshua Landau added the comment: This causes a segmentation fault if any keyword arguments come after a **-unpack. Minimal demo: f(**x, x=x) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2292

[issue2292] Missing *-unpacking generalizations

2015-01-20 Thread Joshua Landau
Joshua Landau added the comment: Just change if (!PyUnicode_Compare(tmp, key)) { when iterating over prior keyword arguments to if (tmp !PyUnicode_Compare(tmp, key)) { since tmp (the argument's name) can now be NULL. -- ___ Python

[issue22451] filtertuple, filterstring and filterunicode don't have optimization for PyBool_Type

2014-09-21 Thread Joshua Landau
Joshua Landau added the comment: That sounds OK to me. It's a bit of a non-issue once you know about it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22451

[issue22451] filtertuple, filterstring and filterunicode don't have optimization for PyBool_Type

2014-09-20 Thread Joshua Landau
New submission from Joshua Landau: All code referred to is from bltinmodule.c, Python 2.7.8: https://github.com/python/cpython/blob/2.7/Python/bltinmodule.c filter implements and optimization for PyBool_Type, making it equivalent to PyNone: # Line 303 if (func == (PyObject

[issue22451] filtertuple, filterstring and filterunicode don't have optimization for PyBool_Type

2014-09-20 Thread Joshua Landau
Joshua Landau added the comment: It's solely a speed thing. I think it was an oversight that the optimisation was only applied to lists. I didn't expect the optimisation to break when applied to tuples. -- ___ Python tracker rep...@bugs.python.org

[issue21642] _ if 1else _ does not compile

2014-06-03 Thread Joshua Landau
Joshua Landau added the comment: Here's a minimal example of the difference: 1e # ... etc ... # SyntaxError: invalid token 1t # ... etc ... # SyntaxError: invalid syntax -- ___ Python tracker rep...@bugs.python.org http

[issue21642] _ if 1else _ does not compile

2014-06-02 Thread Joshua Landau
New submission from Joshua Landau: By the docs, Except at the beginning of a logical line or in string literals, the whitespace characters space, tab and formfeed can be used interchangeably to separate tokens. Whitespace is needed between two tokens only

[issue21593] Clarify re.search documentation first match

2014-05-28 Thread Joshua Landau
New submission from Joshua Landau: The documentation for re.search does not state that it returns the first match. This should be added, or a clarification added if this is implementation-defined. https://docs.python.org/3/library/re.html#re.search --- See also http://stackoverflow.com

[issue21587] Tabs in source

2014-05-26 Thread Joshua Landau
New submission from Joshua Landau: There are tabs in the source: http://hg.python.org/cpython/file/5c8d71516235/Include/listobject.h#l49 I don't really know, but this seems like a release blocker to me ;). -- messages: 219183 nosy: Joshua.Landau priority: normal severity: normal

[issue21547] '!s' formatting documentation bug

2014-05-21 Thread Joshua Landau
New submission from Joshua Landau: In the docs for 2.x about the formatting syntax: https://docs.python.org/2/library/string.html#format-string-syntax it says Two conversion flags are currently supported: '!s' which calls str() on the value, and '!r' which calls repr

[issue2292] Missing *-unpacking generalizations

2013-07-12 Thread Joshua Landau
Joshua Landau added the comment: http://www.python.org/dev/peps/pep-0448/ is out; see what you think. See http://mail.python.org/pipermail/python-ideas/2013-July/021872.html for all the juicy discussion so far. -- nosy: +Joshua.Landau ___ Python

[issue16010] Some Unicode in identifiers improperly rejected

2012-09-23 Thread Joshua Landau
New submission from Joshua Landau: a¹ = None is not valid, even though unicodedata.normalize(NFKC, ¹) == 1. One would expect a¹ = None and a1 = None to be equivalent in this case, as with aⁱ = None and ai = None. I am not sure how many other characters exhibit the same problem. References

[issue15924] 404 link on Python about page

2012-09-11 Thread Joshua Landau
New submission from Joshua Landau: http://www.python.org/about/ section Python plays well with others, last paragraph, link extension modules links to http://www.python.org/doc/ext/intro.html, a 404 page. http://www.python.org/doc/ext/ redirects to http://docs.python.org/extending

[issue13658] Extra clause in class grammar documentation

2011-12-23 Thread Joshua Landau
New submission from Joshua Landau joshua.landau...@gmail.com: Inside the grammar for classes[1], the documentation states that the inheritance list can be of type: ( [argument_list [,] | comprehension] ) The comprehension part seems to be superfluous, especially as it is valid grammar without

[issue13343] Lambda keyword-only argument not updating co_freevars

2011-11-04 Thread Joshua Landau
New submission from Joshua Landau joshua.landau...@gmail.com: When setting defaults to keyword-only arguments in lambdas which are inside non-global scopes, cPython doesn't push the name to it's closure's co_freevars. EXAMPLE: global_variable = None (lambda: (lambda *, keyword_only

[issue13343] Lambda keyword-only argument not updating co_freevars

2011-11-04 Thread Joshua Landau
Joshua Landau joshua.landau...@gmail.com added the comment: Glad to help :) It's made my day. I get to boast at school now! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13343