[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Change by Serhiy Storchaka : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: New changeset 4d3f084c035ad3dfd9f8479886c41b1b1823ace2 by Serhiy Storchaka in branch 'master': bpo-27867: Add a porting guide for PySlice_GetIndicesEx(). (#1973)

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Henk-Jaap Wagenaar added the comment: @serhiy.storchaka: review done. -- nosy: +Henk-Jaap Wagenaar ___ Python tracker ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: Could anyone please make review of the documentation? -- ___ Python tracker ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: PR 1973 adds a porting guide. This should be the last commit for this issue. Please make a review and suggest better wording. -- ___ Python tracker

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Changes by Serhiy Storchaka : -- pull_requests: +2039 ___ Python tracker ___ ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: New changeset b879fe82e7e5c3f7673c9a7fa4aad42bd05445d8 by Serhiy Storchaka in branch 'master': Expand the PySlice_GetIndicesEx macro. (#1023) https://github.com/python/cpython/commit/b879fe82e7e5c3f7673c9a7fa4aad42bd05445d8 New changeset

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: New changeset e41390aca51e4e3eb455cf3b70f5d656a2814db9 by Serhiy Storchaka in branch '2.7': bpo-27867: Expand the PySlice_GetIndicesEx macro. (#1023) (#1046) https://github.com/python/cpython/commit/e41390aca51e4e3eb455cf3b70f5d656a2814db9 --

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Changes by Serhiy Storchaka : -- pull_requests: +1198 ___ Python tracker ___ ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Changes by Serhiy Storchaka : -- pull_requests: +1197 ___ Python tracker ___ ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Changes by Serhiy Storchaka : -- pull_requests: +1199 ___ Python tracker ___ ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: If don't make PySlice_GetIndicesEx a macro when Py_LIMITED_API is not defined, it should be expanded to PySlice_Unpack and PySlice_AdjustIndices. PR 1023 does this for master branch. The patch is generated by Coccinelle's semantic patch. -- Added

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Changes by Serhiy Storchaka : -- pull_requests: +1182 ___ Python tracker ___ ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Changes by Martin Panter : -- nosy: -martin.panter ___ Python tracker ___ ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Changes by Serhiy Storchaka : -- pull_requests: -1038 ___ Python tracker ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Changes by Donald Stufft : -- pull_requests: +1038 ___ Python tracker ___ ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: This issue is left open because it needs to add a porting guide in What's New. See also a problem with breaking ABI in issue29943. -- ___ Python tracker

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Roundup Robot added the comment: New changeset faa1891d4d1237d6df0af4622ff520ccd6768e04 by Serhiy Storchaka in branch '3.6': Issue #27867: Silenced may-be-used-uninitialized warnings after https://github.com/python/cpython/commit/faa1891d4d1237d6df0af4622ff520ccd6768e04 New changeset

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Roundup Robot added the comment: New changeset faa1891d4d1237d6df0af4622ff520ccd6768e04 by Serhiy Storchaka in branch '3.5': Issue #27867: Silenced may-be-used-uninitialized warnings after https://github.com/python/cpython/commit/faa1891d4d1237d6df0af4622ff520ccd6768e04 --

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Roundup Robot added the comment: New changeset 745dda46d2e3e27206bb33188c770e1f6c73766e by Serhiy Storchaka in branch '2.7': Issue #27867: Silenced may-be-used-uninitialized warnings after https://github.com/python/cpython/commit/745dda46d2e3e27206bb33188c770e1f6c73766e New changeset

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Roundup Robot added the comment: New changeset faa1891d4d1237d6df0af4622ff520ccd6768e04 by Serhiy Storchaka in branch 'master': Issue #27867: Silenced may-be-used-uninitialized warnings after https://github.com/python/cpython/commit/faa1891d4d1237d6df0af4622ff520ccd6768e04 New changeset

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Roundup Robot added the comment: New changeset 110ec861e5ea by Serhiy Storchaka in branch '2.7': Issue #27867: Fixed merging error. https://hg.python.org/cpython/rev/110ec861e5ea -- ___ Python tracker

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Roundup Robot added the comment: New changeset d7b637af5a7e by Serhiy Storchaka in branch '3.5': Issue #27867: Silenced may-be-used-uninitialized warnings after https://hg.python.org/cpython/rev/d7b637af5a7e New changeset 17d0cfc64a32 by Serhiy Storchaka in branch '2.7': Issue #27867: Silenced

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Changes by Serhiy Storchaka : Added file: http://bugs.python.org/file46510/PySlice_GetIndicesEx-silence-warnings.patch ___ Python tracker

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: Good point Martin. I missed this because warnings are not emitted in non-debug build and were emitted only once in incremental debug build. Your idea about initializing slicelength in a macro LGTM. -- ___ Python

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Martin Panter added the comment: Not a big deal, but the change produces compiler warnings with GCC 6.1.1: /home/proj/python/cpython/Objects/bytesobject.c: In function ‘bytes_subscript’: /home/proj/python/cpython/Objects/bytesobject.c:1701:13: warning: ‘slicelength’ may be used uninitialized

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Roundup Robot added the comment: New changeset d5590f357d74 by Serhiy Storchaka in branch '2.7': Issue #27867: Replaced function PySlice_GetIndicesEx() with a macro. https://hg.python.org/cpython/rev/d5590f357d74 New changeset 96f5327f7253 by Serhiy Storchaka in branch '3.5': Issue #27867:

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: We can't just add API functions in maintained releases, because it will break the stable ABI. We can use them only when explicitly define the version of API. Proposed patch for 3.6 and 3.7 adds public API functions PySlice_Unpack() and

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: Renamed _PySlice_EvalIndices() to _PySlice_AdjustIndices() and changed its signature. Updated the documentation and python3.def. Fixed yet one bug: implementation-defined behavior with division by negative step. Note that since new functions are used in

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Changes by Serhiy Storchaka : -- assignee: -> serhiy.storchaka versions: +Python 2.7, Python 3.7 ___ Python tracker ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Nick Coghlan added the comment: I think those names (with the leading underscore removed) would be fine as a public API - the fact that PySlice_EvalIndices doesn't take a reference to the slice object seems similar to a static method, where the prefix is there for namespacing reasons, rather

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: Then there is a design question. I believe that after all we should expose these two new functions publicly. And the question is about function names and the order of arguments. Currently signatures are: int _PySlice_Unpack(PyObject *r, Py_ssize_t *start,

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Nick Coghlan added the comment: As in, for arguments that have __index__() methods, do the conversion to a true Python integer eagerly when the slice is built rather than lazily when slice.indices() (or the C-level equivalent) is called? That actually seems like a potentially plausible future

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: Yet one possible solution is to make slice constructor converting arguments to exact ints. This allows to leave user code unchanged. But this is 3.6-only solution of course. I would like to know Mark's thoughts on this. --

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Nick Coghlan added the comment: Nice! The one thing I would suggest double checking with this change is whether or not we have test cases covering ranges with lengths that don't fit into ssize_t. It's been years since I looked at that code, so I don't remember exactly how it currently works,

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Terry J. Reedy added the comment: I like this. Very nice. What I understand is that callers that access PySlice_GetIndicesEx via the header file (included with Python.h) will see the function as a macro. When the macro is expanded, the length expression will be evaluated after any __index__

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: Actually making slicing always working is easier than I expected. Maybe it is even easier than raise an error. PySlice_GetIndicesEx() is split on two functions. First convert slice attributes to Py_ssize_t, then scale them to appropriate range depending on

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: I'm saying that PySlice_GetIndicesEx2 can't just use Py_SIZE. -- ___ Python tracker ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Terry J. Reedy added the comment: FWIW. Py_SIZE is used all over listobject.c. Are you saying that this could be improved? -- ___ Python tracker ___

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: This is a toy example that exposes the problem, but the problem itself is not a toy problem. The key point is that calculating slice indices cause executing Python code and releases GIL. In multithread program a sequence can be changed not in toy __index__

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Terry J. Reedy added the comment: There is really one immediate issue: PySlice_GetIndicesEx in Objects/sliceobject.c takes as inputs a slice object and a sequence length, but not the sequence itself. It starts with "/* this is harder to get right than you might think */" Yep. It assumes

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Serhiy Storchaka added the comment: Even list suffers from this bug if slicing step is not 1. class X: def __index__(self): del a[:] return 1 a = [0] a[:X():2] -- ___ Python tracker

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Terry J. Reedy added the comment: I presume you are suggesting to raise if the length changes. This is similar to raising when a dict is mutated while iterating. Note that we do not do this with mutable sequences. (If the iteration is stopped with out-of-memory error, so be it.) An

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

Changes by Serhiy Storchaka : -- nosy: +mark.dickinson, serhiy.storchaka stage: -> needs patch type: -> crash ___ Python tracker

[issue27867] various issues due to misuse of PySlice_GetIndicesEx

New submission from tehybel: Here I will describe 6 issues with various core objects (bytearray, list) and the array module. Common to them all is that they arise due to a misuse of the function PySlice_GetIndicesEx. This type of issue results in out-of-bounds array indexing which leads to