[issue42246] Implement PEP 626

2020-11-24 Thread Mark Shannon


Change by Mark Shannon :


--
pull_requests: +22384
pull_request: https://github.com/python/cpython/pull/23495

___
Python tracker 
<https://bugs.python.org/issue42246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42373] PEP 626 does not specify behavior of tracing for keywords.

2020-11-23 Thread Mark Shannon


Mark Shannon  added the comment:

https://github.com/python/peps/pull/1722/

--

___
Python tracker 
<https://bugs.python.org/issue42373>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42349] Compiler front-end produces a broken CFG

2020-11-23 Thread Mark Shannon


Change by Mark Shannon :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue42349>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39934] Fatal Python error "XXX block stack overflow" when exception stacks >10

2020-11-17 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset 48a9c0eb2a3304ea64d1b32fdf9db853d5d8c429 by Irit Katriel in 
branch '3.9':
[3.9] bpo-39934: Account for control blocks in 'except' in compiler. (GH-22395) 
(GH-23303)
https://github.com/python/cpython/commit/48a9c0eb2a3304ea64d1b32fdf9db853d5d8c429


--

___
Python tracker 
<https://bugs.python.org/issue39934>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42349] Compiler front-end produces a broken CFG

2020-11-17 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset 266b462238bddec0213effad3650f19c56511e9f by Mark Shannon in 
branch 'master':
bpo-42349: Compiler clean up. More yak-shaving for PEP 626. (GH-23267)
https://github.com/python/cpython/commit/266b462238bddec0213effad3650f19c56511e9f


--

___
Python tracker 
<https://bugs.python.org/issue42349>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42202] Optimize function annotation

2020-11-17 Thread Mark Shannon


Mark Shannon  added the comment:

For top level functions (functions created once) this isn't going to make any 
real difference. There might be a small speedup for function creation, but it 
isn't going to be measurable.

For nested functions with annotations, where many functions are created from a 
single code object, this could be worthwhile.

However, before we add yet another attribute to code objects, I'd like to see 
some evidence of a speedup.

--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue42202>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42373] PEP 626 does not specify behavior of tracing for keywords.

2020-11-16 Thread Mark Shannon


New submission from Mark Shannon :

PEP 626 doesn't cover which keywords are to be traced if they occur on a line 
by themselves.

Some keywords, like `break`, in Python have behavior associated with them.
Others, like `else` are just syntax and don't do anything.
Currently, it is not clear which keywords get traced and which do not.


I'd like to update the PEP to include the following:

Keywords can be broken down into five groups.
1. Must have an attendant expression ( "if", "while", "for", "await")
   These keywords are not traced, since their expressions are.
2. Have an optional expression and perform non-local control flow ( "return", 
"yield", "raise", "except" )
   These keywords are traced. If on a different line from the expression, they 
will generate additional events.
3. Pure syntax ( "else", "try", "finally" )
   These keywords are not traced.
4. Local control flow ( "break", continue", "pass" )
   These keywords are traced
5. Expressions ( "None", "True", "False" )
   Treated like other expressions


Examples:

1. if (
2.test
3. ):

will generate an event for line 2.


1. return (
2.x
3. ):

will generate an event for line 2, then line 1.

1. try:
2.None
3. finally:
4.pass

Will generate an event for line 2, then line 4.


Note that whatever changes, the invariant remains that if a line is in 
`co_lines()` then it will generate trace events, and if it isn't then it won't.

--
assignee: Mark.Shannon
messages: 381114
nosy: Mark.Shannon, nedbat, pablogsal
priority: normal
severity: normal
status: open
title: PEP 626 does not specify behavior of tracing for keywords.
versions: Python 3.10

___
Python tracker 
<https://bugs.python.org/issue42373>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42349] Compiler front-end produces a broken CFG

2020-11-13 Thread Mark Shannon


Change by Mark Shannon :


--
keywords: +patch
pull_requests: +22163
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/23267

___
Python tracker 
<https://bugs.python.org/issue42349>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42349] Compiler front-end produces a broken CFG

2020-11-13 Thread Mark Shannon


New submission from Mark Shannon :

The front-end of the bytecode compiler produces a broken CFG.
A number of "basic-block"s have terminators before their end.
This makes the back-end optimizations unsafe as they rely of a well-formed CFG.

The fix is simple. Insert a check that the CFG is well-formed before doing any 
optimizations, then fix up the front-end.

Once done, we can be more aggressive with optimizations without worrying that 
things will break for no apparent reason.

--
assignee: Mark.Shannon
messages: 380911
nosy: BTaskaya, Mark.Shannon, pablogsal, yselivanov
priority: normal
severity: normal
status: open
title: Compiler front-end produces a broken CFG
type: behavior

___
Python tracker 
<https://bugs.python.org/issue42349>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42246] Implement PEP 626

2020-11-13 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset fd009e606a48e803e7187983bf9a5682e938fddb by Mark Shannon in 
branch 'master':
bpo-42246: Fix memory leak in compiler (GH-23256)
https://github.com/python/cpython/commit/fd009e606a48e803e7187983bf9a5682e938fddb


--

___
Python tracker 
<https://bugs.python.org/issue42246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42246] Implement PEP 626

2020-11-13 Thread Mark Shannon


Change by Mark Shannon :


--
pull_requests: +22153
pull_request: https://github.com/python/cpython/pull/23256

___
Python tracker 
<https://bugs.python.org/issue42246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42246] Implement PEP 626

2020-11-12 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset cc75ab791dd5ae2cb9f6e0c3c5f734a6ae1eb2a9 by Mark Shannon in 
branch 'master':
bpo-42246: Eliminate jumps to exit blocks by copying those blocks. (#23251)
https://github.com/python/cpython/commit/cc75ab791dd5ae2cb9f6e0c3c5f734a6ae1eb2a9


--

___
Python tracker 
<https://bugs.python.org/issue42246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42246] Implement PEP 626

2020-11-12 Thread Mark Shannon


Change by Mark Shannon :


--
pull_requests: +22148
pull_request: https://github.com/python/cpython/pull/23251

___
Python tracker 
<https://bugs.python.org/issue42246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42246] Implement PEP 626

2020-11-12 Thread Mark Shannon


Change by Mark Shannon :


--
pull_requests: +22142
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/23245

___
Python tracker 
<https://bugs.python.org/issue42246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42246] Implement PEP 626

2020-11-12 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset 877df851c3ecdb55306840e247596e7b7805a60a by Mark Shannon in 
branch 'master':
bpo-42246: Partial implementation of PEP 626. (GH-23113)
https://github.com/python/cpython/commit/877df851c3ecdb55306840e247596e7b7805a60a


--

___
Python tracker 
<https://bugs.python.org/issue42246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42294] [C API] Add new C functions with more regular reference counting like PyTuple_GetItemRef()

2020-11-09 Thread Mark Shannon


Mark Shannon  added the comment:

I'm not convinced that this is worth the effort.

The old functions aren't going away, so these additional functions provide no 
real safety.
You can't stop C programmers trading away correctness for some perceived 
performance benefit :(

If we were designing the API from scratch, then this would be a better set of 
functions. But because the old functions remain, it just means we are making 
the API larger.


Please don't add macros, use inline functions.

There seems to be some confusion about borrowed references and stolen 
references in https://pythoncapi.readthedocs.io/bad_api.html#borrowed-references
"Stealing" a reference is perfectly safe. Returning a "borrowed" reference is 
not.

So, don't bother with `PyTuple_SetItemRef()`, as `PyTupleSetItem()` is safe.

--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue42294>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42246] Implement PEP 626

2020-11-03 Thread Mark Shannon


Change by Mark Shannon :


--
nosy: +nedbat

___
Python tracker 
<https://bugs.python.org/issue42246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42246] Implement PEP 626

2020-11-03 Thread Mark Shannon


Mark Shannon  added the comment:

The following code is completely eliminated by the macros.

1. if 0:
2. secret_debugging_code()

PEP 626 says that all executed lines of code must generate trace events,
so we need to emit an instruction for line 1.

Dead code elimination will remove the `secret_debugging_code()`, but leave the 
test. The peephole optimiser can then reduce it to a NOP, but won't eliminate 
it as it is the only instruction for line 1.

--
stage: patch review -> 

___
Python tracker 
<https://bugs.python.org/issue42246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42246] Implement PEP 626

2020-11-02 Thread Mark Shannon


Change by Mark Shannon :


--
keywords: +patch
pull_requests: +22030
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/23113

___
Python tracker 
<https://bugs.python.org/issue42246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42246] Implement PEP 626

2020-11-02 Thread Mark Shannon


New submission from Mark Shannon :

Implementation of PEP 626 requires:

1. Implementation of the new line number table and associated APIs.
2. Removal of BEGIN_DO_NOT_EMIT_BYTECODE and END_DO_NOT_EMIT_BYTECODE from the 
compiler as they do not understand line numbers and may remove lines from the 
bytecode that they shouldn't.
3. Enhance compiler front-end and CFG optimizer to avoid the negative 
performance impact of PEP.
 a) Duplicate the tests in while blocks to avoid the extra jump instruction at 
the end of the loop.
 b) Duplicate and renumber terminator blocks that have no line numbers.

Guaranteeing that f_lineno is correct without hurting performance
-

PEP 626 mandates that the f_lineno attribute of a frame is always correct, even 
after a return or raise, but we don't want to hurt performance.
Since the interpreter ensures that the f_lasti attribute of a frame is always 
correct, we can ensure correctness of f_lineno at zero cost, by ensuring that 
all RETURN_VALUE, RAISE_VARARGS and RERAISE instruction have a non-negative 
line number. Then f_lineno can always be lazily computed from f_lasti.

The front-end generates artificial RERAISEs and RETURN_VALUE that have no line 
number. To give these instructions a valid line number we can take advantage of 
the fact that such terminator blocks (blocks with no successors) can be freely 
duplicated. Once duplicated, each terminator block will have only one 
predecessor and can acquire the line number of the preceding block, without 
causing false line events.

--
assignee: Mark.Shannon
messages: 380231
nosy: Mark.Shannon, pablogsal
priority: normal
severity: normal
status: open
title: Implement PEP 626
type: enhancement
versions: Python 3.10

___
Python tracker 
<https://bugs.python.org/issue42246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42217] compiler: merge co_code and co_lnotab

2020-10-31 Thread Mark Shannon


Mark Shannon  added the comment:

PEP 626 has just been accepted.
Could you please leave this until I have merged in the implementation of PEP 
626 which uses a different line number table format

--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue42217>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42099] Fix reference to ob_type in unionobject.c and ceval

2020-10-27 Thread Mark Shannon


Mark Shannon  added the comment:

Thanks Neil

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue42099>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42099] Fix reference to ob_type in unionobject.c and ceval

2020-10-27 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset 0564aafb71a153dd0aca4b9266dfae9336a4f2cb by Neil Schemenauer in 
branch 'master':
bpo-42099: Fix reference to ob_type in unionobject.c and ceval (GH-22829)
https://github.com/python/cpython/commit/0564aafb71a153dd0aca4b9266dfae9336a4f2cb


--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue42099>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42126] Optimize BUILD_CONST_KEY_MAP for distinct keys

2020-10-23 Thread Mark Shannon


Mark Shannon  added the comment:

It sounds like the root cause is that annotations are repeatedly being 
computed. We should address the underlying issue, IMO.

--

___
Python tracker 
<https://bugs.python.org/issue42126>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41835] Speed up dict vectorcall creation using keywords

2020-10-23 Thread Mark Shannon


Mark Shannon  added the comment:

Could we get a pyperformance benchmark run on this please?

--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue41835>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42126] Optimize BUILD_CONST_KEY_MAP for distinct keys

2020-10-23 Thread Mark Shannon


Mark Shannon  added the comment:

Is this worthwhile?

Statically, BUILD_CONST_KEY_MAP represents 0.14% of all bytecode instructions 
in the standard library, but only 0.03% of the bytecode instructions in 
functions.
Which suggests that dynamically only 1 in 3000 instructions executed is 
BUILD_CONST_KEY_MAP. The implication is that making BUILD_CONST_KEY_MAP more 
complex is just as likely to slow things down as speed them up.

Do you have any evidence that this will make a measurable difference?

--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue42126>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42057] peephole optimizer bug relating to JUMP_IF_NOT_EXC_MATCH

2020-10-22 Thread Mark Shannon


Change by Mark Shannon :


--
pull_requests: +21825
pull_request: https://github.com/python/cpython/pull/22893

___
Python tracker 
<https://bugs.python.org/issue42057>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42057] pytest case which catch exceptions become segfault

2020-10-19 Thread Mark Shannon


Mark Shannon  added the comment:

The test example has no reference to pytest in it.

How are you running it?
Can you produce a segmentation fault when run without pytest?

--

___
Python tracker 
<https://bugs.python.org/issue42057>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue33387] Simplify bytecodes for try-finally, try-except and with blocks.

2020-10-16 Thread Mark Shannon


Mark Shannon  added the comment:

Yes, thanks for pointing that out.

--

___
Python tracker 
<https://bugs.python.org/issue33387>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue33387] Simplify bytecodes for try-finally, try-except and with blocks.

2020-10-16 Thread Mark Shannon


Change by Mark Shannon :


--
pull_requests: +21689
pull_request: https://github.com/python/cpython/pull/22723

___
Python tracker 
<https://bugs.python.org/issue33387>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42033] Seemingly unnecessary complexification of foo(**kw)

2020-10-15 Thread Mark Shannon


Mark Shannon  added the comment:

Have you observed any slowdown or incorrect behaviour?

The 3.8 bytecode looks incorrect to me.
The C-API documentation doesn't prohibit callables from mutating the dictionary 
they receive.
Unless a copy is made, then a callee could mutate `var`.

https://docs.python.org/3/c-api/call.html

--

___
Python tracker 
<https://bugs.python.org/issue42033>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41756] Do not always use exceptions to return result from coroutine

2020-10-11 Thread Mark Shannon


Mark Shannon  added the comment:

What's the difference between removing it and making it properly private (i.e. 
static)?

It's an irrelevant detail whether the code is inlined or in a helper function.

--

___
Python tracker 
<https://bugs.python.org/issue41756>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41756] Do not always use exceptions to return result from coroutine

2020-10-10 Thread Mark Shannon


Mark Shannon  added the comment:

Vladimir,
Thanks for adding PyIter_Send().

Don't forget to remove PyGen_Send() :)

--

___
Python tracker 
<https://bugs.python.org/issue41756>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41670] ceval traces code differently with USE_COMPUTED_GOTOS

2020-09-29 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset 17b5be0c0a3f74141014e06a660f1b5ddb002fec by Mark Shannon in 
branch 'master':
bpo-41670: Remove outdated predict macro invocation. (GH-22026)
https://github.com/python/cpython/commit/17b5be0c0a3f74141014e06a660f1b5ddb002fec


--

___
Python tracker 
<https://bugs.python.org/issue41670>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39934] Fatal Python error "XXX block stack overflow" when exception stacks >10

2020-09-25 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset 02d126aa09d96d03dcf9c5b51c858ce5ef386601 by Mark Shannon in 
branch 'master':
bpo-39934: Account for control blocks in 'except' in compiler. (GH-22395)
https://github.com/python/cpython/commit/02d126aa09d96d03dcf9c5b51c858ce5ef386601


--

___
Python tracker 
<https://bugs.python.org/issue39934>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39934] Fatal Python error "XXX block stack overflow" when exception stacks >10

2020-09-24 Thread Mark Shannon


Change by Mark Shannon :


--
keywords: +patch
pull_requests: +21435
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/22395

___
Python tracker 
<https://bugs.python.org/issue39934>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39934] Fatal Python error "XXX block stack overflow" when exception stacks >10

2020-09-24 Thread Mark Shannon


Mark Shannon  added the comment:

iritkatriel

What error do you see and on what version?

--

___
Python tracker 
<https://bugs.python.org/issue39934>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41845] Promote PyObject_GenericGetDict to the stable API

2020-09-23 Thread Mark Shannon


Mark Shannon  added the comment:

It wasn't removed in 7d95e4072169911b228c9e42367afb5f17fd3db0,
just moved from object.h to dictobject.h
It was still part of the API at that point, I think.

--

___
Python tracker 
<https://bugs.python.org/issue41845>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40941] Merge generator.gi_running and frame executing flag into single frame state

2020-09-23 Thread Mark Shannon


Mark Shannon  added the comment:

Thanks Victor.
Sorry I didn't get round to this sooner

--

___
Python tracker 
<https://bugs.python.org/issue40941>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41810] Consider reintroducing `types.EllipsisType` for the sake of typing

2020-09-21 Thread Mark Shannon


Mark Shannon  added the comment:

I can't resist the pun on typing.
`type(...)` is the least typing :)

More seriously,

>From a general consistency point of view,
if this is to be added, shouldn't `types.NoneType` and 
`types.NotImplementedType` be added as well?

--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue41810>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41756] Do not always use exceptions to return result from coroutine

2020-09-21 Thread Mark Shannon


Mark Shannon  added the comment:

Yury,

Why was the PR merged with a new API function `PyGen_Send`?

I explicitly said that any new API function should *not* start with `PyGen`, 
nor should any function rely on generators and async "coroutines" sharing the 
same memory layout.

If you disagree with me, please say why, don't just merge the PR.

The name `PyGen` is misleading as it can handle coroutines as well as 
generators.
There is no performance advantage to only handling these two types.
Worse, it requires that a `PyCoroObject` can always be cast to a `PyGenObject`, 
preventing better layout of either object in the future.


Would you revert the PR, please.

--

___
Python tracker 
<https://bugs.python.org/issue41756>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41756] Do not always use exceptions to return result from coroutine

2020-09-17 Thread Mark Shannon


Mark Shannon  added the comment:

I agree with Serhiy, that `PyGen_` is a bad prefix.

Unless a function takes generator objects and *only* generators objects, then 
it shouldn't have a `PyGen` prefix.

The API function is the C equivalent of obj.send(val).
The first parameter is an object.

Coroutines do not inherit from generators.
That the the C implementations are so coupled is an unfortunate historical 
accident, and may well be changed.

Regardless of how this is implemented internally, any API function's signature 
should reflect the equivalent Python code.
And please use an enum, not an int, as the return value. It's a huge boon for 
readability.

I would suggest:
`PySendResult PyIter_Send(PyObject *obj, PyObject *arg, PyObject **result);`

--

___
Python tracker 
<https://bugs.python.org/issue41756>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41703] Most bytecode changes are absent from Python 3.9 What's new

2020-09-09 Thread Mark Shannon


Mark Shannon  added the comment:

Thanks for the reminder. Tis' done.

--
stage: patch review -> 

___
Python tracker 
<https://bugs.python.org/issue41703>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41703] Most bytecode changes are absent from Python 3.9 What's new

2020-09-08 Thread Mark Shannon


Mark Shannon  added the comment:

The change note for 33387 incorrectly referred to 32949

--
stage: patch review -> 

___
Python tracker 
<https://bugs.python.org/issue41703>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41703] Most bytecode changes are absent from Python 3.9 What's new

2020-09-08 Thread Mark Shannon


Change by Mark Shannon :


--
keywords: +patch
pull_requests: +21232
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/22151

___
Python tracker 
<https://bugs.python.org/issue41703>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41703] Most bytecode changes are absent from Python 3.9 What's new

2020-09-08 Thread Mark Shannon


Mark Shannon  added the comment:

https://bugs.python.org/issue39320 has a change note:
https://docs.python.org/3.9/whatsnew/changelog.html#python-3-9-0-beta-3

--

___
Python tracker 
<https://bugs.python.org/issue41703>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41703] Most bytecode changes are absent from Python 3.9 What's new

2020-09-08 Thread Mark Shannon


Change by Mark Shannon :


--
assignee:  -> Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue41703>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41670] ceval traces code differently with USE_COMPUTED_GOTOS

2020-08-31 Thread Mark Shannon


Change by Mark Shannon :


--
keywords: +patch
pull_requests: +21126
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/22026

___
Python tracker 
<https://bugs.python.org/issue41670>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41670] ceval traces code differently with USE_COMPUTED_GOTOS

2020-08-31 Thread Mark Shannon


Mark Shannon  added the comment:

A couple of things to fix here.

Firstly, the PREDICTion of POP_BLOCK in FOR_ITER shouldn't be there. POP_BLOCK 
doesn't normally occur after a loop and hasn't since we removed "pseudo 
exceptions" from the interpreter a couple of years ago.

Secondly, there is the issue of PREDICTs skipping tracing.
Either we can make sure that no PREDICTs cross a line boundary, which seems 
error prone, or we add the check for tracing into the PREDICT macro, which 
seems more robust.

--

___
Python tracker 
<https://bugs.python.org/issue41670>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41531] Python 3.9 regression: Literal dict with > 65535 items are one item shorter

2020-08-13 Thread Mark Shannon


Mark Shannon  added the comment:

@hroncok,

How did you discover this issue?

I'd like to clean up the code for creating dictionary literals and it might be 
helpful to know where such huge dictionary literals exist.
I'm guessing that they are used as lookup tables for things like Unicode 
code-point tables, and that they would only include constants.

--

___
Python tracker 
<https://bugs.python.org/issue41531>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41531] Python 3.9 regression: Literal dict with > 65535 items are one item shorter

2020-08-13 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset c51db0ea40ddabaf5f771ea633b37fcf4c90a495 by Pablo Galindo in 
branch 'master':
bpo-41531: Fix compilation of dict literals with more than 0x elements 
(GH-21850)
https://github.com/python/cpython/commit/c51db0ea40ddabaf5f771ea633b37fcf4c90a495


--

___
Python tracker 
<https://bugs.python.org/issue41531>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41463] Avoid duplicating jump information from opcode.py in compile.c

2020-08-04 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset 582aaf19e8b94a70c1f96792197770d604ba0fdf by Mark Shannon in 
branch 'master':
bpo-41463: Generate information about jumps from 'opcode.py' rather than 
duplicating it in 'compile.c' (GH-21714)
https://github.com/python/cpython/commit/582aaf19e8b94a70c1f96792197770d604ba0fdf


--

___
Python tracker 
<https://bugs.python.org/issue41463>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41463] Avoid duplicating jump information from opcode.py in compile.c

2020-08-03 Thread Mark Shannon


Change by Mark Shannon :


--
keywords: +patch
pull_requests: +20858
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/21714

___
Python tracker 
<https://bugs.python.org/issue41463>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41463] Avoid duplicating jump information from opcode.py in compile.c

2020-08-03 Thread Mark Shannon


New submission from Mark Shannon :

opcode.py declares which jumps are relative and which are absolute.
We duplicate that information in compile.c
We should generate lookup tables in opcodes.h and to repeating that information 
in compile.c

--
messages: 374735
nosy: Mark.Shannon
priority: normal
severity: normal
status: open
title: Avoid duplicating jump information from opcode.py in compile.c
type: enhancement

___
Python tracker 
<https://bugs.python.org/issue41463>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41323] Perform "peephole" optimization directly on control-flow graph.

2020-07-30 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset 6e8128f02e1d36e38e5866f9dc36e051caa47bc9 by Mark Shannon in 
branch 'master':
bpo-41323: Perform 'peephole' optimizations directly on the CFG. (GH-21517)
https://github.com/python/cpython/commit/6e8128f02e1d36e38e5866f9dc36e051caa47bc9


--

___
Python tracker 
<https://bugs.python.org/issue41323>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41323] Perform "peephole" optimization directly on control-flow graph.

2020-07-30 Thread Mark Shannon


Change by Mark Shannon :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue41323>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40941] Merge generator.gi_running and frame executing flag into single frame state

2020-07-17 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset cb9879b948a19c9434316f8ab6aba9c4601a8173 by Mark Shannon in 
branch 'master':
bpo-40941: Unify implicit and explicit state in the frame and generator objects 
into a single value. (GH-20803)
https://github.com/python/cpython/commit/cb9879b948a19c9434316f8ab6aba9c4601a8173


--

___
Python tracker 
<https://bugs.python.org/issue40941>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41323] Perform "peephole" optimization directly on control-flow graph.

2020-07-17 Thread Mark Shannon


Change by Mark Shannon :


--
keywords: +patch
pull_requests: +20653
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/21517

___
Python tracker 
<https://bugs.python.org/issue41323>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41323] Perform "peephole" optimization directly on control-flow graph.

2020-07-17 Thread Mark Shannon


New submission from Mark Shannon :

Currently we perform various bytecode improvements as a pass on the code 
objects after generating the code object.

This requires parsing the bytecode to find instructions, recreating the CFG, 
and rewriting the line number table.

If we perform the optimizations directly on the CFG we can avoid all that 
additional work.

This would save hundreds of lines of code and avoid coupling the optimization 
to the line number table format.

--
assignee: Mark.Shannon
messages: 373811
nosy: Mark.Shannon
priority: normal
severity: normal
stage: needs patch
status: open
title: Perform "peephole" optimization directly on control-flow graph.
type: performance

___
Python tracker 
<https://bugs.python.org/issue41323>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29590] Incorrect stack traces when re-entering a generator/coroutine stack via .throw()

2020-07-09 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset 8b33961e4bc4020d8b2d5b949ad9d5c669300e89 by Chris Jerdonek in 
branch 'master':
bpo-29590: fix stack trace for gen.throw() with yield from (#19896)
https://github.com/python/cpython/commit/8b33961e4bc4020d8b2d5b949ad9d5c669300e89


--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue29590>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40941] Merge generator.gi_running and frame executing flag into single frame state

2020-07-09 Thread Mark Shannon


Mark Shannon  added the comment:

issue29590 is unrelated

--

___
Python tracker 
<https://bugs.python.org/issue40941>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40925] Remove redundant macros used for stack manipulation in interpreter

2020-06-13 Thread Mark Shannon


Mark Shannon  added the comment:

Dennis, thanks for doing the benchmarking.
You seem to have removed rather more macros than I suggested,
but it probably won't make much difference to the results.

Since this may have a negative effect on performance and doesn't seem to be 
popular, let's drop it.

For the record, my motivation for this change is for tooling.
Instructions that only use POP, PEEK and PUSH, are much easier to analyze for 
tools like verifiers and optimizers.

Such "nice to have" features aren't worth the slowdown.

--
resolution:  -> wont fix

___
Python tracker 
<https://bugs.python.org/issue40925>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40925] Remove redundant macros used for stack manipulation in interpreter

2020-06-13 Thread Mark Shannon


Change by Mark Shannon :


--
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue40925>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40941] Merge generator.gi_running and frame executing flag into single frame state

2020-06-11 Thread Mark Shannon


Change by Mark Shannon :


--
keywords: +patch
pull_requests: +1
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/20803

___
Python tracker 
<https://bugs.python.org/issue40941>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40941] Merge generator.gi_running and frame executing flag into single frame state

2020-06-11 Thread Mark Shannon


Mark Shannon  added the comment:

This change combines the explicit state in `PyFrameObject.f_excuting` and 
`PyGenObject.gi_running`, and the implicit state in `PyFrameObject.f_stacktop 
== NULL` and `PyFrameObject.f_last == -1` into a single enum field in 
`PyFrameObject`.

Since we no longer need to test `PyFrameObject.f_stacktop == NULL` , The 
`f_stacktop` pointer can be replaced with a `f_stackdepth` integer, which make 
for simpler code when iterating over the stack and avoids the potential hazard 
of NULL pointers.

There are  three benefits to these changes:

1. The code is more robust and, IMO, easier to understand, as all state is now 
explicit.
2. It carries additional information about the state of the frame. Information 
about whether a frame exiting by `return` or by `raise` is available.
3. A modest reduction in size of frame and generator objects.

--

___
Python tracker 
<https://bugs.python.org/issue40941>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40925] Remove redundant macros used for stack manipulation in interpreter

2020-06-11 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset 33faf5c4f43e24766cf567bec89ad4c7f1491ff7 by Dong-hee Na in branch 
'master':
bpo-40925: Remove unused stack macro SET_VALUE (GH-20783)
https://github.com/python/cpython/commit/33faf5c4f43e24766cf567bec89ad4c7f1491ff7


--

___
Python tracker 
<https://bugs.python.org/issue40925>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40941] Merge generator.gi_running and frame executing flag into single frame state

2020-06-10 Thread Mark Shannon


New submission from Mark Shannon :

Generators have a "gi_running" attribute (coroutines have an equivalent 
cr_running flag). Internally frame also has an executing flag.

We use these, plus the test `f_stacktop pointer == NULL`, to determine what 
state a generator or frame is in.

It would be much cleaner to maintain a single f_state field in the frame to 
track the state of a frame, reducing the change of ambiguity and error.

The possible states of a frame are:
CREATED -- Frame exists but has not been executed at all
SUSPENDED -- Frame has been executed, and has been suspended by a yield
EXECUTING -- Frame is being executed
RETURNED -- Frame has completed by a RETURN_VALUE instruction
RAISED -- Frame has completed as a result of an exception being raised
CLEARED -- Frame has been cleared, either by explicit call to clear or by the 
GC.

--
assignee: Mark.Shannon
components: Interpreter Core
messages: 371194
nosy: Mark.Shannon
priority: normal
severity: normal
stage: needs patch
status: open
title: Merge generator.gi_running and frame executing flag into single frame 
state
type: performance

___
Python tracker 
<https://bugs.python.org/issue40941>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40925] Remove redundant macros used for stack manipulation in interpreter

2020-06-10 Thread Mark Shannon


Mark Shannon  added the comment:

I'm proposing that we remove the nine macros above; the eleven listed, except 
TOP() and SECOND().
They can be replaced by POP(), PUSH() and PEEK() without lost of functionality 
or performance.

--

___
Python tracker 
<https://bugs.python.org/issue40925>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40925] Remove redundant macros used for stack manipulation in interpreter

2020-06-09 Thread Mark Shannon


New submission from Mark Shannon :

Currently, there are fourteen macros provided for stack manipulation.
Along with the fundamental, POP(), PUSH() and PEEK(n) the following eleven are 
also provided:

TOP()  
SECOND()   
THIRD()
FOURTH()   
SET_TOP(v) 
SET_SECOND(v)  
SET_THIRD(v)   
SET_FOURTH(v)  
SET_VALUE(n, v)
STACK_GROW(n)
STACK_SHRINK(n) 

These are unnecessary as compilers have long understood the sequences of 
pointer increments and decrements that is generated by using POPs and PUSHes.

See https://godbolt.org/z/Htw-2k which shows GCC generating exactly the same 
code from just POP, PUSH and PEEK as from direct access to items on the stack.

Removing the redundant macros would make the code easier to reason about and 
analyze.
TOP() and SECOND() are used quite heavily and are trivial to convert to PEEK by 
tools, so should probably be left alone.

Notes:

I'm ignoring the stack debugging macros here, they aren't a problem.
The EMPTY() macro is only used twice, so might as well be replaced with 
STACK_LEVEL == 0.
Sadly, there is no "maintainability/code quality" selection for "type" of 
issue, so I've chosen "performance" :(

--
components: Interpreter Core
keywords: easy (C)
messages: 371087
nosy: Mark.Shannon
priority: normal
severity: normal
stage: needs patch
status: open
title: Remove redundant macros used for stack manipulation in interpreter
type: performance

___
Python tracker 
<https://bugs.python.org/issue40925>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40521] [subinterpreters] Make free lists and unicode caches per-interpreter

2020-06-08 Thread Mark Shannon


Mark Shannon  added the comment:

I'd be interested to see if you can get more consistent results.

Performance of modern hardware is very sensitive to memory layout, so some sort 
of address randomization might be needed to remove artifacts of layout.
It is possible that the objects on the free lists for telco are better aligned 
with cache lines, or fit is cache better in some way.
And conversely, in chameleon, objects fit cache in a worse way.
Just a guess, of course.

Thanks for trying to get some benchmark results.

--

___
Python tracker 
<https://bugs.python.org/issue40521>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40521] [subinterpreters] Make free lists and unicode caches per-interpreter

2020-06-05 Thread Mark Shannon


Mark Shannon  added the comment:

I'm worried about the performance impact of these changes, especially as many 
of the changes haven't been reviewed.

Have you done any performance analysis or tests of the cumulative effect of all 
these changes?

--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue40521>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40830] Certain uses of dictionary unpacking raise TypeError

2020-05-31 Thread Mark Shannon


Change by Mark Shannon :


--
keywords:  -patch

___
Python tracker 
<https://bugs.python.org/issue40830>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40830] Certain uses of dictionary unpacking raise TypeError

2020-05-31 Thread Mark Shannon


Change by Mark Shannon :


--
keywords: +patch
pull_requests: +19796
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/20553

___
Python tracker 
<https://bugs.python.org/issue40830>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29882] Add an efficient popcount method for integers

2020-05-31 Thread Mark Shannon


Mark Shannon  added the comment:

Why are calling a population count method "bit_count()"?
That seems likely to cause confusion with "bit_length()".

I might reasonable expect that 0b1000.bit_count() be 4, not 1. It does have 4 
bits.
Whereas 0b1000.population_count() is clearly 1.

I have no objection to adding this method, just the choice of name.

--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue29882>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40830] Certain uses of dictionary unpacking raise TypeError

2020-05-30 Thread Mark Shannon


Mark Shannon  added the comment:

Thanks for the report.
I'm looking in to it.

--
assignee:  -> Mark.Shannon
priority: normal -> release blocker
stage:  -> needs patch

___
Python tracker 
<https://bugs.python.org/issue40830>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40728] UnboundLocalError as a result of except statement variable re-assignment

2020-05-22 Thread Mark Shannon


Mark Shannon  added the comment:

Oleksandr,
Are you saying that the current behaviour is incorrect, in the sense that it 
disagrees with the documentation?
Or are you saying that the current behaviour is undesirable?

--

___
Python tracker 
<https://bugs.python.org/issue40728>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39511] [subinterpreters] Per-interpreter singletons (None, True, False, etc.)

2020-05-15 Thread Mark Shannon


Mark Shannon  added the comment:

Those numbers are for code without immortal objects.
They don't apply in this case, as the branch misprediction rate would rise.

--

___
Python tracker 
<https://bugs.python.org/issue39511>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40545] Expose _PyErr_GetTopmostException

2020-05-07 Thread Mark Shannon


Mark Shannon  added the comment:

Why do you need to specify a `PyThreadState`?

--

___
Python tracker 
<https://bugs.python.org/issue40545>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40421] [C API] Add getter functions for PyFrameObject and maybe move PyFrameObject to the internal C API

2020-05-07 Thread Mark Shannon


Mark Shannon  added the comment:

"maybe a few setter functions as well"
Please don't add any setter functions, that would a major change to the VM and 
would need a PEP.

Also, could you remove PyFrame_GetLastInstr(PyFrameObject *frame)?
The only purpose of `f_lasti` is to get the line number and that can be done 
directly via `PyFrame_GetLineNumber(PyFrameObject *frame)`

Perhaps Stefan can tell us why Cython needs to access the internals of the 
frame object.

--
nosy: +Mark.Shannon, scoder

___
Python tracker 
<https://bugs.python.org/issue40421>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40545] Expose _PyErr_GetTopmostException

2020-05-07 Thread Mark Shannon


Mark Shannon  added the comment:

What's wrong with `PyErr_GetExcInfo()` for accessing the current exception?

Victor, would you revert https://github.com/python/cpython/pull/19978 until we 
decide what to do. Thanks.

--

___
Python tracker 
<https://bugs.python.org/issue40545>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40228] Make setting line number in frame more robust.

2020-05-07 Thread Mark Shannon


Change by Mark Shannon :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue40228>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40228] Make setting line number in frame more robust.

2020-04-29 Thread Mark Shannon


Mark Shannon  added the comment:


New changeset 57697245e1deafdedf68e5f21ad8890be591efc0 by Mark Shannon in 
branch 'master':
bpo-40228: More robust frame.setlineno. (GH-19437)
https://github.com/python/cpython/commit/57697245e1deafdedf68e5f21ad8890be591efc0


--

___
Python tracker 
<https://bugs.python.org/issue40228>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40255] Fixing Copy on Writes from reference counting

2020-04-20 Thread Mark Shannon


Mark Shannon  added the comment:

On 20/04/2020 2:33 pm, Steve Dower wrote:
> 
> Steve Dower  added the comment:
> 
>> I would expect that the negative impact on branch predictability would 
>> easily outweigh the cost of the memory write (A guaranteed L1 hit)
> 
> If that were true then Spectre and Meltdown wouldn't have been so interesting 
> :)

I really don't follow your logic here.

> 
> Pipelining processors are going to speculatively execute both paths, and will 
> skip the write much more quickly than by doing it, and meanwhile nobody 
> should have tried to read the value so it hasn't had to block for that path. 
> I'm not aware of any that detect no-op writes and skip synchronising across 
> cores - the dirty bit of the cache line is just set unconditionally.

Processors only execute the one path, speculatively or otherwise, that's 
why branch prediction is so important.
(And is important for the Spectre attack. If what you said were true, 
the Spectre attack wouldn't need to pre-bias the branch predictor.)
I'm assuming a modern Intel processor.

> 
> Benchmarking already showed that the branching version is faster. It's 
> possible that "refcount += (refcount & IMMORTAL) ? 0 : 1" could generate 
> different code (should be mov,test,lea,cmovz rather than mov,and,add,mov or 
> mov,and,jz,add,mov), but it's totally reasonable for a branch to be faster 
> than unconditionally modifying memory.

It hasn't been shown that it is faster. I'm not claiming that it isn't, 
just that I haven't seen the evidence.

> 
> --
> 
> ___
> Python tracker 
> <https://bugs.python.org/issue40255>
> ___
>

--

___
Python tracker 
<https://bugs.python.org/issue40255>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40255] Fixing Copy on Writes from reference counting

2020-04-20 Thread Mark Shannon


Mark Shannon  added the comment:

Eddie,
How did you compare the branching vs. non-branching implementations?  I have 
read the code in the PR. What is the non-branching version that you used to 
compare it against?

Why not use the sign bit as the immortal bit? It simplifies the test 
considerably, making it faster, and doesn't need any assembly code.

Are you suggesting that making a set of common objects immortal will make 
things faster? Have you tested it?
(I would expect that the negative impact on branch predictability would easily 
outweigh the cost of the memory write (A guaranteed L1 hit).

--

___
Python tracker 
<https://bugs.python.org/issue40255>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40255] Fixing Copy on Writes from reference counting

2020-04-16 Thread Mark Shannon


Mark Shannon  added the comment:

Do you have more details on your comparison?
I'm somewhat puzzled how a version that does no more work and has no jumps is 
slower.

The memory hit is not immediate, but making the object header immutable 
prevents changes like 
https://github.com/markshannon/cpython/commit/c73407e7b5d1a0fc794d55c6bcc91cfdc958f6c4
which would potentially save the two word per object overhead that the cyclic 
GC currently imposes.

--

___
Python tracker 
<https://bugs.python.org/issue40255>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40255] Fixing Copy on Writes from reference counting

2020-04-16 Thread Mark Shannon


Mark Shannon  added the comment:

A big -1 to this.

You are asking the whole world to take a hit on both performance and memory 
use, in order to save Instagram memory.

The PR uses the term "immortal" everywhere. There is only one reference to 
copy-on-write in a comment. Yet this issue about making object headers 
immutable.
Immortality and object header immutability are not the same.

If object header immutability is to be a requirement, that needs a PEP.

If it is not requirement, but immortality is, then make the obvious improvement 
of changing the branchy code

if (!(obj->refcnt & IMMORTAL_BIT)) {
   obj->refcnt++;
}

to the branchless

obj->refcnt += ((obj->refcnt _BIT) != 0)

Immortality has advantages because it allows saturating reference counting and 
thus smaller object headers, but it is not the same as making the object header 
immutable.

--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue40255>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40225] recursive call within generator expression is O(depth)

2020-04-11 Thread Mark Shannon


Change by Mark Shannon :


--
keywords: +patch
pull_requests: +18828
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19473

___
Python tracker 
<https://bugs.python.org/issue40225>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40225] recursive call within generator expression is O(depth)

2020-04-11 Thread Mark Shannon


Mark Shannon  added the comment:

The problem is that generators always raise an exception, even when they 
terminate normally.
They don't in 2.7 and didn't prior to 
https://github.com/python/cpython/commit/1f7ce62bd61488d5d721896a36a1b43befab88b5#diff-23c87bfada1d01335a3019b9321502a0

--

___
Python tracker 
<https://bugs.python.org/issue40225>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40228] Make setting line number in frame more robust.

2020-04-08 Thread Mark Shannon


Change by Mark Shannon :


--
keywords: +patch
pull_requests: +18791
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19437

___
Python tracker 
<https://bugs.python.org/issue40228>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40228] Make setting line number in frame more robust.

2020-04-08 Thread Mark Shannon


New submission from Mark Shannon :

Debuggers are allowed to change the line number of the currently executing 
frame. Regardless of whether this is sensible or not, the current 
implementation rather fragile.

The code makes various assumptions about the layout of the bytecode that may 
not be true in the future, and I suspect, are not true now.

We should use a more brute-force approach of computing the exception stack for 
the whole function and then searching for a safe place to jump to. This is not 
only more robust it allows more jumps.

For example, it is safe to jump from one exception handler to another.
It may not be sensible, but it is safe.

--
components: 2to3 (2.x to 3.x conversion tool)
messages: 365990
nosy: Mark.Shannon
priority: normal
severity: normal
status: open
title: Make setting line number in frame more robust.

___
Python tracker 
<https://bugs.python.org/issue40228>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40222] "Zero cost" exception handling

2020-04-08 Thread Mark Shannon


New submission from Mark Shannon :

C++ and Java support what is known as "zero cost" exception handling.
The "zero cost" refers to the cost when no exception is raised. There is still 
a cost when exceptions are thrown.

The basic principle is that the compiler generates tables indicating where 
control should be transferred to when an exception is raised. When no exception 
is raised, there is no runtime overhead.

(C)Python should support "zero cost" exceptions.


Now that the bytecodes for exception handling are regular (meaning that their 
stack effect can be statically determined) it is possible for the bytecode 
compiler to emit exception handling tables.

Doing so would have two main benefits.
1. "try" and "with" statements would be faster (and "async for", but that is an 
implementation detail).
2. Calls to Python functions would be faster as frame objects would be 
considerably smaller. Currently each frame carries 240 bytes of overhead for 
exception handling.

--
assignee: Mark.Shannon
messages: 365974
nosy: Mark.Shannon
priority: normal
severity: normal
status: open
title: "Zero cost" exception handling
type: performance

___
Python tracker 
<https://bugs.python.org/issue40222>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

2020-03-31 Thread Mark Shannon


Mark Shannon  added the comment:

You can't end up with a growing set of keys.
The problem is to do with the *order* of insertion, not the number of keys.

--

___
Python tracker 
<https://bugs.python.org/issue40116>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

2020-03-30 Thread Mark Shannon


Mark Shannon  added the comment:

Just to clarify.

class AlwaysShared:

   opt = DEFAULT

   def __init__(self, attr, optional=None):
   self.attr = attr
   if optional:
   self.opt = optional

class SometimesShared:

   opt = DEFAULT

   def __init__(self, attr, optional=None):
   if optional:
   self.opt = optional
   self.attr = attr

The class AlwaysShared always has shared keys, whereas the class 
SometimesShared is not shared if `optional` is True for first instantiation, 
but False for a later instantiation.

--

___
Python tracker 
<https://bugs.python.org/issue40116>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

2020-03-30 Thread Mark Shannon


Mark Shannon  added the comment:

Indeed it shouldn't.
How is that relevant?

--

___
Python tracker 
<https://bugs.python.org/issue40116>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

2020-03-30 Thread Mark Shannon


New submission from Mark Shannon :

The current implementation of dicts prevents keys from being shared when the 
order of attribute differs from the first instance created.
This can potentially use a considerably larger amount of memory than expected.

Consider the class:

class C:

   opt = DEFAULT

   def __init__(self, attr, optional=None):
   if optional:
   self.opt = optional
   self.attr = attr

This is a reasonable way to write a class, but has unpredictable memory use.
In the attached example, per-instance dict size goes from 104 bytes to 232 
bytes when sharing is prevented.

The language specification says that the dicts maintain insertion order, but 
the wording implies that this only to explicit dictionaries, not instance 
attribute or other namespace dicts.

Either we should allow key sharing in these cases, or clarify the documentation.

--
components: Interpreter Core
files: compact_dict_prevents_key_sharing.py
messages: 365319
nosy: Mark.Shannon, inada.naoki
priority: normal
severity: normal
stage: test needed
status: open
title: Regression in memory use of shared key dictionaries for "compact dicts"
type: behavior
Added file: 
https://bugs.python.org/file49013/compact_dict_prevents_key_sharing.py

___
Python tracker 
<https://bugs.python.org/issue40116>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2020-03-27 Thread Mark Shannon


Mark Shannon  added the comment:

Just to add my 2 cents.

I think this a bad idea and is likely to be unsafe.
Having interpreters interfering with each other's objects is almost certain to 
lead to race conditions.

IMO, objects should *never* be shared across interpreters (once interpreters 
are expected to be able to run in parallel).

Any channel between two interpreters should consist of two objects, one per 
interpreter. The shared memory they use to communicated can be managed by the 
runtime. Likewise for inter-interpreter locks. The lock itself should be 
managed by the runtime, with each interpreter having its own lock object with a 
handle on the shared lock.

--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue33608>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39511] [subinterpreters] Per-interpreter singletons (None, True, False, etc.)

2020-03-17 Thread Mark Shannon


Mark Shannon  added the comment:

Having two CPUs write to the same cache line is a well known performance 
problem. There's nothing special about CPython here.

The proper name for it seems to be "cache line ping-pong", but a search for 
"false sharing" might be more informative.

--

___
Python tracker 
<https://bugs.python.org/issue39511>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

2020-03-17 Thread Mark Shannon


Mark Shannon  added the comment:

Even if `_PyRuntime` ends up as just a list of interpreters and doesn't 
disappear completely, it won't be used anything like as much as it is now.

Many of the functions that it getting passed to will no longer need it, so why 
bother passing it now?

--

___
Python tracker 
<https://bugs.python.org/issue36710>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

2020-03-17 Thread Mark Shannon


Mark Shannon  added the comment:

Instead of passing `_PyRuntimeState` around everywhere, why not just let it 
disappear in time.

Currently `_PyRuntimeState` manages "global" state, mainly the GIL and some 
config.
Once the GIL has been migrated to the sub-interpreters, the config part can be 
factored out and `_PyRuntimeState` can just disappear.

--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue36710>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39511] [subinterpreters] Per-interpreter singletons (None, True, False, etc.)

2020-03-17 Thread Mark Shannon


Mark Shannon  added the comment:

Consider the case where a thread that doesn't hold the GIL attempts to get a 
reference on `None`.

The problem with having a single immortal `None`, is that it will cause data 
cache thrashing as two different CPUs modify the refcount on the shared `None` 
object.
Each subinterpreter needs its own distinct `None`.

`None` could be made immortal, it just can't be shared between sub-interpreters.

--
nosy: +Mark.Shannon

___
Python tracker 
<https://bugs.python.org/issue39511>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



  1   2   3   4   >