[issue46657] Add mimalloc memory allocator
Neil Schemenauer added the comment: My preference would be for --with-mimalloc=yes in an upcoming release. For platforms without the required stdatomic.h stuff, they can manually specify --with-mimalloc=no. That will make them aware that a future release of Python might no longer build (if mimalloc is no longer optional). A soft-landing for merging nogil is not a good enough reason to merge mimalloc, IMHO. nogil may never be merged. There should be some concrete and immediate advantage to switch to mimalloc. The idea of using the "heap walking" to improve is cyclic GC is not concrete enough. It's just an idea at this point. I think the (small) performance win could be enough of a reason to merge. This seems to be the most recent benchmark: https://gist.github.com/pablogsal/8027937b71cd30f175ef7c885d3e There is also the long-term maintenance issue. So far, mimalloc upstream has been responsive. The mimalloc code is not so huge or complicated that we couldn't maintain it (if for some reason it gets abandoned upstream). However, I think we would prefer to maintain obmalloc rather than mimalloc, all else being equal. Abandonment by the upstream seems fairly unlikely. So, I'm not too concerned about maintenance. -- ___ Python tracker <https://bugs.python.org/issue46657> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46657] Add mimalloc memory allocator
Neil Schemenauer added the comment: Thanks, I'm indeed interested. Most credit goes to Christian for advancing this. For the missing stdatomic.h, would it be appropriate to have an autoconfig check for it? Can just disable mimalloc if it doesn't exist. -- ___ Python tracker <https://bugs.python.org/issue46657> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45561] Run smelly.py and multissltests.py from $(srcdir)
Change by Neil Schemenauer : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue45561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38006] Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit
Neil Schemenauer added the comment: Closing since I believe the bug is fixed and there is an appropriate unit test. -- assignee: -> nascheme resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue38006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45561] Run smelly.py and multissltests.py from $(srcdir)
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +27415 pull_request: https://github.com/python/cpython/pull/29138 ___ Python tracker <https://bugs.python.org/issue45561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45561] Run smelly.py and multissltests.py from $(srcdir)
New submission from Neil Schemenauer : Some makefile rules don't work if you build in a separate folder. -- messages: 404671 nosy: nascheme priority: normal severity: normal stage: patch review status: open title: Run smelly.py and multissltests.py from $(srcdir) type: behavior ___ Python tracker <https://bugs.python.org/issue45561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45521] obmalloc radix tree typo in code
Change by Neil Schemenauer : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue45521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45521] obmalloc radix tree typo in code
Neil Schemenauer added the comment: New changeset 1cdac61065e72db60d26e03ef9286d2743d7000e by Miss Islington (bot) in branch '3.10': bpo-45521: Fix a bug in the obmalloc radix tree code. (GH-29051) (GH-29122) https://github.com/python/cpython/commit/1cdac61065e72db60d26e03ef9286d2743d7000e -- ___ Python tracker <https://bugs.python.org/issue45521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45521] obmalloc radix tree typo in code
Neil Schemenauer added the comment: New changeset 311910b31a4bd94dc79298388b7cb65ca5546438 by Neil Schemenauer in branch 'main': bpo-45521: Fix a bug in the obmalloc radix tree code. (GH-29051) https://github.com/python/cpython/commit/311910b31a4bd94dc79298388b7cb65ca5546438 -- ___ Python tracker <https://bugs.python.org/issue45521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45521] obmalloc radix tree typo in code
Neil Schemenauer added the comment: I have not yet been able to reproduce methane's crash. My guess it it's not related. An explanation of what I think the impact of this bug is: The radix tree is used to determine if memory is from obmalloc or from the system malloc (i.e return value from address_in_range()). WIth ADDRESS_BITS set to 48, we ignore the top 16 bits of addresses. The next 10 bits are supposed to be the index into the top level node array for the radix tree. Due to the bug, we mask those and only use the bottom 8 of those 10. So, if you have virtual addresses that span more than that 8 bit range, we will index into the wrong node. That means address_in_range() could give the wrong answer. Which means you might try to free memory with the wrong malloc. I think this is likely to be triggered only if you allocate a massive amount of memory, like 70 TB. However, triggering it would depend on how the kernel maps virtual memory to the Python process. I.e. there might be a wierd OS that gives pages at 0x7f00 and then right after pages at 0x3f00. -- ___ Python tracker <https://bugs.python.org/issue45521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45526] Set ADDRESS_BITS to 64 for obmalloc radix tree
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +27329 pull_request: https://github.com/python/cpython/pull/29062 ___ Python tracker <https://bugs.python.org/issue45526> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45526] Set ADDRESS_BITS to 64 for obmalloc radix tree
New submission from Neil Schemenauer : Given this feedback: https://github.com/python/cpython/pull/14474/files#r725488766 it is perhaps not so safe to assume that only the lower 48 bits of virtual addresses are significant. I had the idea that Go made similar assumptions but now I'm not sure it does. There is also a comment in this article about 5-level page tables in Linux: https://lwn.net/Articles/717293/ https://lwn.net/Articles/717300/ I.e. that Linux does not allocate virtual address space above 47-bit by default. Setting ADDRESS_BITS to 64 is safer and the performance impact seems small. The virtual memory size of a small Python process goes up a little. Resident set size doesn't significantly change. I think the pyperformance changes are just noise. The pyperformance attached file shows the 3.10 branch with ADDRESS_BITS set to 48 and to 64. -- assignee: methane files: pyperf-compare-addr-64.txt messages: 404317 nosy: methane, nascheme priority: normal severity: normal stage: patch review status: open title: Set ADDRESS_BITS to 64 for obmalloc radix tree Added file: https://bugs.python.org/file50370/pyperf-compare-addr-64.txt ___ Python tracker <https://bugs.python.org/issue45526> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45521] obmalloc radix tree typo in code
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +27322 pull_request: https://github.com/python/cpython/pull/29051 ___ Python tracker <https://bugs.python.org/issue45521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45521] obmalloc radix tree typo in code
New submission from Neil Schemenauer : There is a typo in the radix tree obmalloc code, spotted by Inada Naoki. -#define MAP_TOP_MASK (MAP_BOT_LENGTH - 1) +#define MAP_TOP_MASK (MAP_TOP_LENGTH - 1) This should be fixed both in the main branch and in 3.10.x. -- assignee: methane components: Interpreter Core messages: 404268 nosy: methane, nascheme priority: high severity: normal stage: patch review status: open title: obmalloc radix tree typo in code versions: Python 3.10, Python 3.11 ___ Python tracker <https://bugs.python.org/issue45521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45020] Freeze all modules imported during startup.
Neil Schemenauer added the comment: [Larry] > The one thing I guess I never mentioned is that building and working with the > prototype was frightful; it had both Python code and C code, and it was > fragile and hard to get working. I took Larry's PR and did a fair amount of cleanup on it to make the build less painful and fragile. My branch is fairly easy to re-build. The major downsides remaining are that you couldn't update .py files and have them used (static ones take priority) and the generated C code is quite large. I didn't make any attempt to work on the serializer, other than to make it work with an alpha version of Python 3.10. https://github.com/nascheme/cpython/tree/static_frozen It was good enough to pass nearly(?) all tests and I did some profiling. It helped reduce startup time quite a bit. -- ___ Python tracker <https://bugs.python.org/issue45020> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of PyObject_GC_UnTrack() with the trashcan C API
Neil Schemenauer added the comment: Another small correction, _PyType_IS_GC() checks only the type flag (Py_TPFLAGS_HAVE_GC). _PyObject_IS_GC() checks both the type flag and calls tp_is_gc(), if it exists. tp_is_gc() is wart, IMHO, and it would be nice if we can kill it off so only the type flag is needed. That's a topic for a different issue though. -- ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44897] Integrate trashcan mechanism into _Py_Dealloc
Change by Neil Schemenauer : -- nosy: +pablogsal, tim.peters, vstinner ___ Python tracker <https://bugs.python.org/issue44897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44897] Integrate trashcan mechanism into _Py_Dealloc
Neil Schemenauer added the comment: Based on some testing, I think an extra type slot is not worth the extra complication. I made some small improvements to _Py_Dealloc and now the performance seems a bit better. Basically, I expanded _PyObject_IS_GC() to put the most common branches first. See pypref-trashcan2.txt for benchmark. Overall I think might be a bit slower (less than 1%?) but we have gained trashcan protection for the dealloc of all GC objects. If we are okay with the performance, here are other questions to answer: A) is it safe to untrack GC objects before calling tp_dealloc? B) is it safe to have PyObject_CallFinalizerFromDealloc() track objects that are resurrected? We might have to look at 3rd party extensions to decide on those. I.e. maybe in theory it is not safe but in practice there is no extension code that would actually break. -- Added file: https://bugs.python.org/file50223/pyperf-trashcan2.txt ___ Python tracker <https://bugs.python.org/issue44897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44897] Integrate trashcan mechanism into _Py_Dealloc
Change by Neil Schemenauer : Added file: https://bugs.python.org/file50220/perf-annotate-trash.txt ___ Python tracker <https://bugs.python.org/issue44897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44897] Integrate trashcan mechanism into _Py_Dealloc
Neil Schemenauer added the comment: As I suspected, the performance impact is significant (although pretty small). Based on Linux perf, it looks like the extra test+branch in _Py_Dealloc adds about 1% overhead. pyperformance shows something similar, see attached reports (pypref-trashcan.txt and perf-annotate-trash.txt). An idea I've been trying is to add another type slot, e.g. tp_call_dealloc. It could be set by PyType_Ready(). It would be called by the Py_DECREF macro on refcnt going to zero. If the object is non-GC and Py_TRACE_REFS is off, can make tp_call_dealloc actually be the tp_dealloc pointer. If the type has the GC flag, point tp_call_dealloc to a _Py_Dealloc version that checks tp_is_gc and does the trashcan stuff. Unfortunately, all types don't have PyType_Ready() called on them. So, we cannot rely on tp_call_dealloc being set correctly. -- Added file: https://bugs.python.org/file50219/pyperf-trashcan.txt ___ Python tracker <https://bugs.python.org/issue44897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of PyObject_GC_UnTrack() with the trashcan C API
Neil Schemenauer added the comment: > I think in any case we should benchmark this because this will affect *all* > GC types if I understand correctly and the TS mechanism had shown slowdowns > before We definitely need to benchmark. I would guess that adding trashcan protection to all GC types would not be a performance issue. We already had the macros for some performance critical ones (frame, tuple, list, dict). The performance hit will likely come as a result of adding a branch that uses _PyType_IS_GC() to the DECREF code path. It means any time an object hits zero refcount, we call _PyType_IS_GC() on it. Previously, we would just call tp_dealloc. Because of PEP 442, _PyType_IS_GC() checks not only a type flag but might also call tp_is_gc. So, benchmarks will need to be done. We might get a small win because the trashcan logic can be better optimized now that it's in a single complied unit. Small correction for my explaination above: if tp_dealloc gets called mutiple times because of the trashcan, it is the code before the BEGIN macro that gets called mutiple times. -- ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of PyObject_GC_UnTrack() with the trashcan C API
Neil Schemenauer added the comment:
> The problem of PyObject_GC_UnTrack() is just the most visible effect of the
> trashcan mecanism: tp_dealloc can be called twice, and this is *not* expected
> by the tp_dealloc API.
The fact that Py_TRASHCAN_BEGIN and Py_TRASHCAN_END can cause tp_dealloc to be
called more than once is not a huge issue, IMHO. The statements executed more
than once are the ones before or after the trashcan macros. If you use them
properly, the multiple calls doesn't cause issues. The big trap is
to use _PyObject_GC_UNTRACK before Py_TRASHCAN_BEGIN. It is not safe to call
that macro multiple times. Anywhere you see Py_TRASHCAN_BEGIN and
_PyObject_GC_UNTRACK before it, that's a bug.
The point of this PR is to make the macros harder to use incorrectly. If there
is no need to call untrack, Py_TRASHCAN_BEGIN and Py_TRASHCAN_END can normally
be the first and last lines of the tp_dealloc function (declarations can be
inside). Then, the fact that tp_dealloc is actually called more than once by
the
trashcan doesn't cause an issue.
tp_dealloc can get called more than once for another reason. If the object is
resurrected by tp_finalize or tp_del, then tp_dealloc can be called again.
It's pretty mind bending to try to understand all of implications of that. Most
of the ugliness is inside PyObject_CallFinalizerFromDealloc(). The type with
the finalizer has to be careful not to put the object in an invalid state
before it gets resurrected.
> Putting trashcan mecanism outside tp_dealloc can allow to make sure that
> tp_dealloc is called exactly once. _Py_Dealloc() sounds like a good
> candidate, but I didn't check if it's the only way to call tp_dealloc. Are
> there other places where tp_dealloc is called *directly*?
tp_dealloc is called from a few more places, as you found. As for providing
C-stack exhaustion protection via the trashcan, I think we are only interested
in places were tp_dealloc is called as a result of a DECREF. And I think that
only goes through _Py_Dealloc().
I suppose it would be possible to blow up the C stack via a call chain like:
subtype_dealloc -> basedealloc -> subtype_dealloc -> basedealloc -> ...
I think you would have to make a long chain of subclasses. Seems unlikely in
real code so I'm not worried about trying to fix that.
> Using military grade regex, I found the following functions calling
> tp_dealloc:
>
> grep 'dealloc[) ]*([a-zA-Z]\+)' */*.c
>
> * _Py_Dealloc() obviously
> * _PyTrash_thread_destroy_chain()
> * subtype_dealloc()
> * Modules/_testcapimodule.c: check_pyobject_freed_is_freed() <= unit test, it
> can be ignored
>
> So if we move the trashcan usage inside functions, 3 functions must be
> modified:
>
> * _Py_Dealloc()
> * _PyTrash_thread_destroy_chain()
> * subtype_dealloc()
Take a look at bpo-44897. It implements the _Py_Dealloc approach. You can see
what needs to be modified. I removed the Py_TRASHCAN_BEGIN/Py_TRASHCAN_END
macros as well, just because they are not needed anymore. _PyObject_GC_UNTRACK
calls inside tp_dealloc methods need to be removed because _Py_Dealloc already
does the untrack. For external extensions, they must be using
PyObject_GC_UnTrack() and so that's safe (object is already untracked and so it
does nothing).
I think the big change is calling tp_dealloc methods with the object already
untracked. If an extension did something like:
mytype_dealloc(PyObject *self)
{
...
assert(PyObject_GC_IsTracked(self));
...
}
That would break after PR 27738, at least as it's currently written.
The other issue I'm not quite sure about is that
PyObject_CallFinalizerFromDealloc insists that GC objects are tracked when that
function is called. I think it is okay to just call _PyObject_GC_TRACK on the
ones that are not tracked, to ensure that. Maybe Tim Peters will jump in and
correct the errors of my thinking.
--
nosy: +tim.peters
___
Python tracker
<https://bugs.python.org/issue44881>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of PyObject_GC_UnTrack() with the trashcan C API
Neil Schemenauer added the comment: I wrote a proof-of-concept as bpo-44897. PR 27718 (this issue) might a slightly better performance but I like the other one better because it doesn't expose so much implementation detail to extension types. Either of them require careful review before merging. -- ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44897] Integrate trashcan mechanism into _Py_Dealloc
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +26218 pull_request: https://github.com/python/cpython/pull/27738 ___ Python tracker <https://bugs.python.org/issue44897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44897] Integrate trashcan mechanism into _Py_Dealloc
New submission from Neil Schemenauer : This is a WIP/proof-of-concept of doing away with Py_TRASHCAN_BEGIN and Py_TRASHCAN_END and instead integrating the functionality into _Py_Dealloc. There are a few advantages: - all container objects have the risk of overflowing the C stack if a long reference chain of them is created and then deallocated. So, to be safe, the tp_dealloc methods for those objects should be protected from overflowing the stack. - the Py_TRASHCAN_BEGIN and Py_TRASHCAN_END macros are hard to understand and a bit hard to use correctly. Making the mechanism internal avoids confusion. The code can be slightly simplified as well. This proof-of-concept seems to pass tests but it will need some careful review. The exact rules related to calling GC Track/Untrack are subtle and this changes things a bit. I.e. tp_dealloc is called with GC objects already untracked. For 3rd party extensions, they are calling PyObject_GC_UnTrack() and so I believe they should still work. The fact that PyObject_CallFinalizerFromDealloc() wants GC objects to definitely be tracked is a bit of a mystery to me (there is an assert to check that). I changed the code to track objects if they are not already tracked but I'm not sure that's correct. There could be a performance hit, due to the _PyType_IS_GC() test that was added to the _Py_Dealloc() function. For non-GC objects, that's going to be a new branch and I'm worried it might hurt a bit. OTOH, maybe it's just in the noise. Profiling will need to be done. -- components: Interpreter Core messages: 399433 nosy: nascheme priority: normal severity: normal stage: patch review status: open title: Integrate trashcan mechanism into _Py_Dealloc type: enhancement ___ Python tracker <https://bugs.python.org/issue44897> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of PyObject_GC_UnTrack() with the trashcan C API
Neil Schemenauer added the comment: I was thinking about this more today and I think the better fix is to actually build the trashcan mechanism into _Py_Dealloc(). Requiring that types opt-in to the trashcan mechanism by using the trashcan macros is not ideal. First, the trashcan macros are a bit tricky to use correctly. Second, every "container" type is potentially a part of a long ref chain and could blow up the stack on deallocation (i.e. triggered from DECREF). So, for correctness/robustness, every type that supports cyclic GC should get trashcan-style deallocation. We would have to find a way to do this without incurring a (significant) performance hit. Also, it would have to be done without breaking C extensions. Ideally they would get trashcan-style deallocation without any source code changes. I'm not sure if that can be done but I'm hopeful it's possible. -- ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of GC_UNTRACK with TRASHCAN
Neil Schemenauer added the comment:
Since C99 is now allowed, perhaps we should suggest that declarations go after
Py_TRASHCAN_BEGIN, e.g.
mytype_dealloc(mytype *p)
{
Py_TRASHCAN_BEGIN(p, mytype_dealloc)
... declarations go here ...
... The body of the deallocator goes here, including all calls ...
... to Py_DECREF on contained objects. ...
Py_TRASHCAN_END// there should be no code after this
}
The only dealloc method that doesn't fit this pattern is subtype_dealloc() and
that's because the object might not be a GC object.
Given the pattern, it occurs to me that that _Py_Dealloc() could do the
trashcan begin/end work. However, the authors of the dealloc methods would
have to realize the special rules of the trashcan (e.g. no returns from the
dealloc method body). Also, there would be some overhead added to
_Py_Dealloc() to check if the trashcan is supported. E.g. checking a type flag.
Another idea would be to add a new slot for the method, e.g. tp_dealloc_trash.
Then, _Py_Dealloc() could check if it exists and if so do the trashcan
begin/end dance around it. That would still add some overhead to _Py_Dealloc()
so I think the current macros are the best approach.
--
___
Python tracker
<https://bugs.python.org/issue44881>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of GC_UNTRACK with TRASHCAN
Neil Schemenauer added the comment: For examples of bugs caused by forgetting the untrack calls, see bpo-31095. -- ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of GC_UNTRACK with TRASHCAN
Neil Schemenauer added the comment: Extensions that call PyObject_GC_UnTrack before calling Py_TRASHCAN_BEGIN will still work, they will just take a very minor performance hit. I don't think it is worth the trouble to introduce new macros for that reason. Extensions that really care about performance can wrap the call in a Python version ifdef. There is an issue if someone writes and tests their extension with the new API, i.e. without having the explicit PyObject_GC_UnTrack() call in their dealloc method. If they compile with an older Python, they likely get a crash. If they compile with asserts enable, they would get an assert fail in _PyTrash_thread_deposit_object, i.e.: _PyObject_ASSERT(op, !_PyObject_GC_IS_TRACKED(op)); I guess that's an argument for new macros. -- stage: patch review -> needs patch ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of GC_UNTRACK with TRASHCAN
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +26201 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/27718 ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33930] Segfault with deep recursion into object().__dir__
Neil Schemenauer added the comment: I'm thinking that the explicit call to PyObject_GC_UnTrack should be made unnecessary by integrating it into the trashcan code. That way, we avoid someone else running into this kind of bug in the future. See bpo-44881. -- nosy: +nascheme ___ Python tracker <https://bugs.python.org/issue33930> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44881] Consider integration of GC_UNTRACK with TRASHCAN
New submission from Neil Schemenauer : The fix for bpo-33930 includes a somewhat mysterious comment: // The Py_TRASHCAN mechanism requires that we be able to // call PyObject_GC_UnTrack twice on an object. I wonder if we can just integrate the untrack into the body of the trashcan code. Then, the explicit call to untrack in the dealloc function body can be removed. That removes the risk of incorrectly using the macro version. I suspect the reason this was not done originally is because the original trashcan mechanism did not use the GC header pointers to store objects. Now, any object that uses the trashcan *must* be a GC object. -- messages: 399356 nosy: nascheme priority: normal severity: normal stage: needs patch status: open title: Consider integration of GC_UNTRACK with TRASHCAN type: enhancement ___ Python tracker <https://bugs.python.org/issue44881> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43384] Include regen-stdlib-module-names in regen-all
Change by Neil Schemenauer : -- resolution: -> rejected stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue43384> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44523] weakref.proxy documentation might be outdated
Neil Schemenauer added the comment: It seems to me the old behaviour (don't forward hash) was done for good reasons. If the referent might go away, it is not valid to use it as a dict key since the hash and eq result changes. If it can't go away, what reason is there to use a weakref and not a direct reference? -- ___ Python tracker <https://bugs.python.org/issue44523> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44331] Generate static PyCodeObjects for faster startup
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +25161 pull_request: https://github.com/python/cpython/pull/26571 ___ Python tracker <https://bugs.python.org/issue44331> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44331] Generate static PyCodeObjects for faster startup
New submission from Neil Schemenauer : Note: This is a proof of concept and not ready for merging as is. This is based on 'frozen_modules' from Jeethu Rao , via Larry Hastings. Larry's git branch was: g...@github.com:larryhastings/cpython.git not_another_use_of_the_word_frozen Usage: - Compile Python as normal - Run "make regen-freeze-startup" to re-generate Python/frozenmodules_code.c - Compile Python a second time Changes from Larry's branch: - Move static C code generation tool to Tools/freeze2 - Move _serializer to Modules - Rebase on Python 3.10.0b1 - determine startup modules by running sys.executable - use importlib.util.find_spec() to get code objects - fix ref-counting when setting __path__ - put static frozen modules in frozen_code_objects dict - reduce set of "bad" modules as it seems only _collections_abc needs exclusion - fix the is_frozen_package() and is_frozen() functions to find static frozen code It's not passing all unit tests yet but I'm somewhat hopeful there are no deep problems. Porting the changes from 3.6 to 3.8 and then to 3.10 was not too horrible. There was a few changes to PyGC_Head, to the PyCodeObject structure and to the runtime initialization process. That gives me some hope that it wouldn't be too burdensome to maintain this in the long-term. Mostly it would be updating _serialize.c to keep up with PyCodeObject changes. Based on benchmarking with 3.8, this gives a decent speedup for startup of a trival program, e.g. python -c "True". I measure it as taking 76% of the time. The savings are mostly in marshal.c but there is also some importlib/filesystem overhead that's removed. -- messages: 395243 nosy: nascheme priority: low severity: normal stage: patch review status: open title: Generate static PyCodeObjects for faster startup type: performance ___ Python tracker <https://bugs.python.org/issue44331> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42972] [C API] Heap types (PyType_FromSpec) must fully implement the GC protocol
Change by Neil Schemenauer : -- nosy: +nascheme ___ Python tracker <https://bugs.python.org/issue42972> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43593] pymalloc is not aware of Memory Tagging Extension (MTE) and crashes
Neil Schemenauer added the comment: > If MTE is actually being used, system software assigns "random" values to 4 > of the higher-order bits. Oh, interesting. Two ideas about handling that: we could change our assertion check to be different on ARM platforms that we know have a certain size physical address space. Probably just turn off that high-bits check. Second idea, we could change the radix tree to not assume high address bits are unused. That's trickier to do without performance or memory usage degradations. I have a work-in-progress patch that adds a cache on top of the radix tree lookup. It looks like that cache can be made to have a pretty high hit rate. Based on a small amount of testing, the radix tree lookup for address_in_range() only happens about 1% of the time. If that approach works, we could add another layer to the tree and handle the full 64-bit address space. Based on my wip testing, my benchmark was showing about equal performance with the cache to without. So, no benefit to offset the increase in code complexity. Handling the MTE high bits tricks might enough to justify the cache addition. -- versions: +Python 3.9 -Python 3.8 ___ Python tracker <https://bugs.python.org/issue43593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37368] test_asyncio: test_create_server_ssl_match_failed() failed on s390x SLES 3.x and logged an unraisable exception
Neil Schemenauer added the comment:
It seems to not be specific to S390, same kind of failure on the PPC64LE RHEL8
LTO + PGE 3.x tester:
Exception ignored in:
Traceback (most recent call last):
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/sslproto.py",
line 321, in __del__
self.close()
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/sslproto.py",
line 316, in close
self._ssl_protocol._start_shutdown()
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/sslproto.py",
line 590, in _start_shutdown
self._abort()
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/sslproto.py",
line 731, in _abort
self._transport.abort()
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/selector_events.py",
line 680, in abort
self._force_close(None)
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/selector_events.py",
line 731, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/base_events.py",
line 745, in call_soon
self._check_closed()
File
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/asyncio/base_events.py",
line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
--
___
Python tracker
<https://bugs.python.org/issue37368>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37368] test_asyncio: test_create_server_ssl_match_failed() failed on s390x SLES 3.x and logged an unraisable exception
Neil Schemenauer added the comment:
Seems this failure might be back. At least, the traceback looks quite similar
to me.
The buildbot failed with this:
heads/master:85b6b70589, Mar 29 2021, 22:53:15
0:05:26 load avg: 3.95 [426/427] test_tokenize passed (56.0 sec) -- running:
test_asyncio (44.8 sec)
0:05:35 load avg: 3.34 [427/427/1] test_asyncio failed (env changed) (54.1 sec)
Warning -- Unraisable exception
Exception ignored in:
Traceback (most recent call last):
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py",
line 321, in __del__
self.close()
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py",
line 316, in close
self._ssl_protocol._start_shutdown()
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py",
line 590, in _start_shutdown
self._abort()
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py",
line 731, in _abort
self._transport.abort()
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/selector_events.py",
line 680, in abort
self._force_close(None)
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/selector_events.py",
line 731, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/base_events.py",
line 745, in call_soon
self._check_closed()
File
"/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/base_events.py",
line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
--
nosy: +nascheme
resolution: out of date ->
stage: resolved ->
status: closed -> open
versions: +Python 3.10 -Python 3.9
___
Python tracker
<https://bugs.python.org/issue37368>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43593] pymalloc is not aware of Memory Tagging Extension (MTE) and crashes
Neil Schemenauer added the comment: I've merged PR 14474 so you can just test with an up-to-date "main" branch and see if that fixes the problem. I would expect it should fix the problem since with the radix tree arena tracking, no memory unsanitary behaviour remains. -- nosy: +nascheme ___ Python tracker <https://bugs.python.org/issue43593> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37448] obmalloc: radix tree for tracking arena address ranges
Change by Neil Schemenauer : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue37448> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37448] obmalloc: radix tree for tracking arena address ranges
Neil Schemenauer added the comment: New changeset 85b6b70589c187639aeebc560d67e9cc04abb4d8 by Neil Schemenauer in branch 'master': bpo-37448: Use radix tree for pymalloc address_in_range(). (GH-14474) https://github.com/python/cpython/commit/85b6b70589c187639aeebc560d67e9cc04abb4d8 -- ___ Python tracker <https://bugs.python.org/issue37448> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43445] Add frozen modules to sys.stdlib_module_names
Neil Schemenauer added the comment: Not sure the proper place to discuss this but I wonder if putting this stdlib module names list in the executable is the best idea. The list of available stdlib modules could change after compiling Python. I understand you don't want to dynamically crawl the library path the build the list. That's too slow. However, is there a really strong reason to embed it in the Python executable? Did you consider generating a .py module, containing the list. E.g. "_stdlib_modules.py" inside the lib folder. Then, you can have site.py or some similar startup logic import that module and assign it to sys.stdlib_module_names. -- nosy: +nascheme ___ Python tracker <https://bugs.python.org/issue43445> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42246] Implement PEP 626 -- Precise line numbers for debugging
Neil Schemenauer added the comment: New changeset 87ec26b812e9c4095c017dc60f246eda37b83ab2 by Neil Schemenauer in branch 'master': bpo-43372: Use _freeze_importlib for regen-frozen. (GH-24759) https://github.com/python/cpython/commit/87ec26b812e9c4095c017dc60f246eda37b83ab2 -- ___ Python tracker <https://bugs.python.org/issue42246> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Change by Neil Schemenauer : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue43372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Neil Schemenauer added the comment: New changeset 87ec26b812e9c4095c017dc60f246eda37b83ab2 by Neil Schemenauer in branch 'master': bpo-43372: Use _freeze_importlib for regen-frozen. (GH-24759) https://github.com/python/cpython/commit/87ec26b812e9c4095c017dc60f246eda37b83ab2 -- ___ Python tracker <https://bugs.python.org/issue43372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Change by Neil Schemenauer : -- assignee: -> nascheme ___ Python tracker <https://bugs.python.org/issue43372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42246] Implement PEP 626 -- Precise line numbers for debugging
Change by Neil Schemenauer : -- pull_requests: +23532 pull_request: https://github.com/python/cpython/pull/24759 ___ Python tracker <https://bugs.python.org/issue42246> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Change by Neil Schemenauer : -- pull_requests: +23531 pull_request: https://github.com/python/cpython/pull/24759 ___ Python tracker <https://bugs.python.org/issue43372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43382] github CI blocked by the Ubuntu CI with an SSL error
Neil Schemenauer added the comment: It seems it is enough to make a new commit. The CI seems to re-base and re-run the PR. At least, it worked on two of my PRs. -- ___ Python tracker <https://bugs.python.org/issue43382> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43384] Include regen-stdlib-module-names in regen-all
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +23494 pull_request: https://github.com/python/cpython/pull/24713 ___ Python tracker <https://bugs.python.org/issue43384> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43384] Include regen-stdlib-module-names in regen-all
New submission from Neil Schemenauer : While I was fixing the regen-frozen issue, I noticed it seems unnecessary to have regen-stdlib-module-names separate from regen-all. Maybe Victor knows why it needs to be separate. If it doesn't need to be separate, the CI scripts can be slightly simplified. -- components: Build messages: 388003 nosy: nascheme priority: normal severity: normal stage: patch review status: open title: Include regen-stdlib-module-names in regen-all type: enhancement ___ Python tracker <https://bugs.python.org/issue43384> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43382] github CI blocked by the Ubuntu CI with an SSL error
Neil Schemenauer added the comment: I think it may be related to bpo-41561. There is a bug in the Ubuntu tracker as well: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1899878 I agree with the temporary fix to use "ubuntu-18.04" for CI testing. -- nosy: +nascheme ___ Python tracker <https://bugs.python.org/issue43382> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42246] Implement PEP 626 -- Precise line numbers for debugging
Change by Neil Schemenauer : -- pull_requests: +23492 pull_request: https://github.com/python/cpython/pull/24714 ___ Python tracker <https://bugs.python.org/issue42246> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Change by Neil Schemenauer : -- pull_requests: +23491 pull_request: https://github.com/python/cpython/pull/24714 ___ Python tracker <https://bugs.python.org/issue43372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43381] add small test for frozen module line number table
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +23490 pull_request: https://github.com/python/cpython/pull/24712 ___ Python tracker <https://bugs.python.org/issue43381> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43381] add small test for frozen module line number table
New submission from Neil Schemenauer : In bug #43372, we didn't notice that the code for the __hello__ module was not re-generated. Things seems to be okay but the line number table was corrupted. It seems a good idea to add a small test to ensure that doesn't happen again. I marked the test as CPython implementation specific. -- components: Tests messages: 387989 nosy: nascheme priority: low severity: normal stage: patch review status: open title: add small test for frozen module line number table type: enhancement ___ Python tracker <https://bugs.python.org/issue43381> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +23484 stage: -> patch review pull_request: https://github.com/python/cpython/pull/24708 ___ Python tracker <https://bugs.python.org/issue43372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42246] Implement PEP 626 -- Precise line numbers for debugging
Change by Neil Schemenauer : -- nosy: +nascheme nosy_count: 7.0 -> 8.0 pull_requests: +23485 pull_request: https://github.com/python/cpython/pull/24708 ___ Python tracker <https://bugs.python.org/issue42246> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43372] ctypes: test_frozentable fails when make regen-frozen
Neil Schemenauer added the comment:
I believe the line table format got changed but the frozen code didn't get
re-generated. If you try to call co_lines() on the __hello__ code, Python
crashes.
>>> import __hello__
Hello world!
>>> co = __hello__.__spec__.loader.get_code('__hello__')
>>> co.co_linetable
b'\x04\x01'
>>> list(co.co_lines())
python: ../Objects/codeobject.c:1185: PyLineTable_NextAddressRange: Assertion
`!at_end(range)' failed.
My PR re-generates the code and fixes the test. Perhaps I should also add a
test to exercise co_lines() on the frozen code object.
--
___
Python tracker
<https://bugs.python.org/issue43372>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42212] Check if generated files are up-to-date in Github Actions
Change by Neil Schemenauer : -- nosy: +nascheme nosy_count: 3.0 -> 4.0 pull_requests: +23486 pull_request: https://github.com/python/cpython/pull/24708 ___ Python tracker <https://bugs.python.org/issue42212> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37448] obmalloc: radix tree for tracking arena address ranges
Change by Neil Schemenauer : Added file: https://bugs.python.org/file49834/perf_compare_radix4x.txt ___ Python tracker <https://bugs.python.org/issue37448> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37448] obmalloc: radix tree for tracking arena address ranges
Change by Neil Schemenauer : Added file: https://bugs.python.org/file49833/perf_compare_noradix.txt ___ Python tracker <https://bugs.python.org/issue37448> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43288] test_importlib failure due to missing skip() method
Neil Schemenauer added the comment: New changeset 44fe32061d60f4bd9c4fa48c24e3e4ba26033dba by Neil Schemenauer in branch '3.9': [3.9] bpo-43288: Fix bug in test_importlib test. (GH-24616) https://github.com/python/cpython/commit/44fe32061d60f4bd9c4fa48c24e3e4ba26033dba -- ___ Python tracker <https://bugs.python.org/issue43288> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43288] test_importlib failure due to missing skip() method
Change by Neil Schemenauer : -- pull_requests: +23397 pull_request: https://github.com/python/cpython/pull/24616 ___ Python tracker <https://bugs.python.org/issue43288> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43288] test_importlib failure due to missing skip() method
Neil Schemenauer added the comment: New changeset 84f7afe65c29330f3ff8e318e054b96554a2dede by Neil Schemenauer in branch 'master': Fix failed merge of bpo-43288. (GH-24614) https://github.com/python/cpython/commit/84f7afe65c29330f3ff8e318e054b96554a2dede -- ___ Python tracker <https://bugs.python.org/issue43288> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43288] test_importlib failure due to missing skip() method
Change by Neil Schemenauer : -- pull_requests: +23395 pull_request: https://github.com/python/cpython/pull/24614 ___ Python tracker <https://bugs.python.org/issue43288> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43288] test_importlib failure due to missing skip() method
Neil Schemenauer added the comment: New changeset 50288aa8c955f66ab67a7dadf250ea5f4238eb67 by Neil Schemenauer in branch 'master': bpo-43288: Fix bug in test_importlib test. (GH-24612) https://github.com/python/cpython/commit/50288aa8c955f66ab67a7dadf250ea5f4238eb67 -- ___ Python tracker <https://bugs.python.org/issue43288> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39791] New `files()` api from importlib_resources.
Change by Neil Schemenauer : -- nosy: +nascheme nosy_count: 9.0 -> 10.0 pull_requests: +23392 pull_request: https://github.com/python/cpython/pull/24612 ___ Python tracker <https://bugs.python.org/issue39791> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39791] New `files()` api from importlib_resources.
Change by Neil Schemenauer : -- nosy: +nascheme, nascheme nosy_count: 9.0 -> 10.0 pull_requests: +23392, 23393 pull_request: https://github.com/python/cpython/pull/24612 ___ Python tracker <https://bugs.python.org/issue39791> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43288] test_importlib failure due to missing skip() method
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +23391 stage: -> patch review pull_request: https://github.com/python/cpython/pull/24612 ___ Python tracker <https://bugs.python.org/issue43288> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43288] test_importlib failure due to missing skip() method
New submission from Neil Schemenauer :
The FileSystem class is missing a skip() method. If the file system doesn't
support Unicode filenames, the test crashes.
File "/home/nas/src/cpython/Lib/test/test_importlib/fixtures.py", line 221,
in unicode_filename
self.skip("File system does not support non-ascii.")
AttributeError: 'FileSystem' object has no attribute 'skip'
I'm running tests inside a Debian 32-bit container and for some reason the
test.support.FS_NONASCII variable is not set.
--
components: Tests
messages: 387471
nosy: nascheme
priority: normal
severity: normal
status: open
title: test_importlib failure due to missing skip() method
type: behavior
___
Python tracker
<https://bugs.python.org/issue43288>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16954] Add docstrings for ElementTree module
Change by Neil Schemenauer : -- resolution: -> fixed stage: needs patch -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue16954> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39448] Add regen-frozen makefile target
Change by Neil Schemenauer : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue39448> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39448] Add regen-frozen makefile target
Neil Schemenauer added the comment: New changeset ffa55d21b4a86ad8b4a43a9f597151e526541130 by Neil Schemenauer in branch 'master': bpo-39448: Add regen-frozen makefile target. (GH-18174) https://github.com/python/cpython/commit/ffa55d21b4a86ad8b4a43a9f597151e526541130 -- ___ Python tracker <https://bugs.python.org/issue39448> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42608] Installation failed from source code on Debian ([307/416] test_socket)
Neil Schemenauer added the comment: Maybe something to do the configuration of the machine, so that TCP/UDP socket networking if failing? Two things to try: - give the "python" process a SIGINT signal and you should get a traceback showing where it is hanging. Or you could use "gdb" and attach to the process (assuming you know how to use gdb). - configure without PGO (i.e. don't specify "--enable-optimizations" to conifgure). That will result in a slightly slower build but the build will not require running of the tests. -- nosy: +nascheme ___ Python tracker <https://bugs.python.org/issue42608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42307] make install must not copy python.o into $prefix/lib/python3.10/config-3.10-x86_64-linux-gnu/
Neil Schemenauer added the comment: I think the comments are correct in that it is used to create a new statically linked interpreter that includes a user provided extension module. We could include python.o inside the libpythonXX.a file but then I think it breaks if you are embedding (e.g. linking .a to your own .o that has a main function). It seems probable that no one uses python.o anymore but my preference would be to not remove it unless we are going to completely remove support for statically linking. Nothing is really hurt by having that .o file in the install and the feature still works as originally intended. It would be good to add a comment in the makefile, describing the possible use for that file. -- ___ Python tracker <https://bugs.python.org/issue42307> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34106] Add --with-module-config= to 'configure' script
Change by Neil Schemenauer : -- resolution: -> out of date stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue34106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34704] Do not access ob_type directly, introduce Py_TP
Neil Schemenauer added the comment: This has been resolved using Py_TYPE() and Py_SET_TYPE(). -- resolution: -> out of date stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue34704> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27377] Add socket.fdtype()
Neil Schemenauer added the comment: I believe this is not needed anymore. Closing. -- resolution: -> rejected stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue27377> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39573] [C API] Make PyObject an opaque structure in the limited C API
Neil Schemenauer added the comment: Correction: I think you *cannot* have it both ways. -- ___ Python tracker <https://bugs.python.org/issue39573> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39573] [C API] Make PyObject an opaque structure in the limited C API
Neil Schemenauer added the comment: > I don't understand the rationale for this change in depth, but > does the benefit outweigh (yet another) backwards incompatibility? I think you can have it both ways. Do you want a C API that is stable over a long period of CPython releases or do you want to continue to be able to look deep (i.e. non opaque PyObject*) into CPython implementation internals? During the sprint last week, we talked about how to provide a compatible API, similar to what Pypy cpyext does. It would be possible to provide a (nearly) fully compatible API with the approach. It could get quite painful for CPython to maintain such a thing however. E.g. cpyext has proxy objects (to maintain CPython compatible structure layouts) but keeping those proxies in sync with the internal VM object structures is expensive and tricky. Certainly making PyObject opaque is going to break some 3rd party code. Making it opaque for the non-limited API is not an option IMHO because it breaks too much 3rd party code. Is making it opaque for the limited C API going to break too much code? Maybe, I don't know. Thanks for pointing out pycurl and breezy. -- ___ Python tracker <https://bugs.python.org/issue39573> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42099] Fix reference to ob_type in unionobject.c and ceval
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +21784 pull_request: https://github.com/python/cpython/pull/22829 ___ Python tracker <https://bugs.python.org/issue42099> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42099] Fix reference to ob_type in unionobject.c and ceval
New submission from Neil Schemenauer : It is great that access to ob_type has been cleaned up to use an access macro. There are two spots that still need fixing. I think we should do something to help avoid this kind of thing slipping into the code in the future. E.g. a special build flag that renames the ob_type member would work. I'll make a separate bug for that. -- assignee: vstinner components: Interpreter Core messages: 379155 nosy: nascheme, vstinner priority: normal severity: normal stage: patch review status: open title: Fix reference to ob_type in unionobject.c and ceval type: enhancement ___ Python tracker <https://bugs.python.org/issue42099> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40255] Fixing Copy on Writes from reference counting
Neil Schemenauer added the comment: I resurrected an old thread on Discourse that seems quite relevant to this PR: https://discuss.python.org/t/switching-from-refcounting-to-libgc/1641/35?u=nas -- ___ Python tracker <https://bugs.python.org/issue40255> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40255] Fixing Copy on Writes from reference counting
Neil Schemenauer added the comment: > immutability vs immutable object headers Sorry, I meant to write "immortal vs immutable". -- ___ Python tracker <https://bugs.python.org/issue40255> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40255] Fixing Copy on Writes from reference counting
Neil Schemenauer added the comment: A few thoughts First, the PR is well done. The changes are pretty minimal and are well localized. Thanks Eddie for writing clean code and responding to review requests. My feeling is that, in current state, this still should not get merged or at least not get enabled by default. The main consideration is what is the cost of this new feature vs what is the benefit. If the immortalize heap function is not called by default, all Python users pay a performance hit for this feature. Even after recent PR improvements, if you don't immortalize the heap, Python is slower. Only a small fraction of Python users will benefit from this feature. I think the "pre-fork, CoW exploiting" application architecture is more common than some people would realize (it is a successful way to achieve multi-processing with CPython, making good use of multiple CPU cores). However, I'm pretty sure it is used by a very tiny fraction of all Python programs. If we do immortalize by default (e.g. on Python startup), the semantics of the refcnt field are subtly changed. I suspect some programs will be broken as a result of this change. A clue about what might go wrong comes from the unicode object code. E.g. the resize_inplace() function in unicodeobject.c. It is possible that a non-trivial amount of other 3rd party code will make assumptions about refcnt that will be broken by this change. That code could be fixed but it is a matter of cost vs benefit. If it is disabled by default with a build flag we have an ABI compatibility problem. I.e. incref/decref are inlined functions/macros. Also, as mentioned above, these kinds of build options tend to make maintenance and testing harder. The frozen GC generation did not have these kinds of issues. If people don't use it, there is basically no cost. I think it was a good idea to merge the frozen generation feature. There is a small amount of ongoing maintenance cost but that's acceptable. Regarding Mark's comment about immutability vs immutable object headers, I would frame the goal of the PR differently. Like the GC frozen generation, this immutable instance change is about providing a way to opt-out of Python's default garbage collection mechanism. The frozen generation was to opt-out of the mark-and-sweep style cyclic collection. This PR is about opting-out of the default Python reference counting. Put that way, could we consider this PR as an incremental step in a process of making it possible to have a non-reference count based GC for CPython? E.g. perhaps we could eventually have a mixture of mark-and-sweep and reference counted GC, in order to support the handle (HPy) API. If we want to move towards that, we want CPython code to stop making assumptions about the refcnt field, like the unicodeobject.c file currently does. I think pitching the PR in that way makes it more compelling. Merge it not because it helps a tiny class of Python applications. Instead, merge it because it helps find unwarranted assumptions about how the Python GC works. Once we get bad assumptions cleaned up in 3rd party code, we have more of chance of pursuing an alternative GC strategy for CPython. -- ___ Python tracker <https://bugs.python.org/issue40255> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40255] Fixing Copy on Writes from reference counting
Neil Schemenauer added the comment: Eddie mentions in the PR about using memory arenas to contain immortal objects. I think it could be worth investigating further. With the current PR, the immortal status is dependent on the value of the refcnt field of the object. Using immortal arenas might be faster because you can check if an object is immortal just based on its address (no data dependency on refcnt value). The fastest would be to create an immortal block as part of the BSS (uninitialized data). Then, within incref/decref you use the location and size of the immortal arena (compile time constants) to test if the object is immortal. Maybe could just check if the high bits of an object address match a constant mask (if immortal region is aligned and is power of 2 in size). Slightly slower but more flexible would be to make the immortal arena size and location global variables. That way, you can set the size of the region on startup. Also would be more flexible in terms of ABI compatibility. That would introduce one or two global loads within incref/decref but those values would almost certainly be in L1 cache. Even more flexible would be to use a memory map to mark which arenas are immortal. See my radix tree implementation for obmalloc: https://bugs.python.org/issue37448 I would guess the radix tree lookups are too expensive to put in incref/decref. Should probably test that though. I had started doing an experiment with the arena approach before I noticed Eddie's comment about it. I would like to see his version. Here is a sketch of mine (not working yet): - change new_arena() to initially allocate from an "immortal memory" region. There are multiple ways to allocate that (BSS/uninitialized data, aligned_alloc(), etc). - add a _PyMem_enable_immortal() call to switch obmalloc from using immortal arenas to regular ones. Need to mess with some obmalloc data structures to switch arenas (usedpools, usable_arenas, unused_arena_objects). - change incref/decref to check if immortal status has been enabled and if object address falls within immortal region. If so, incref/decref don't do anything. By default, the immortal arenas could be enabled on Python startup. Call _PyMem_enable_immortal after startup but before running user code. There could be a command-line option to disable the automatic call to _PyMem_enable_immortal() so that users like Instagram can do their pre-fork initialization before calling it. Next step down the rabbit-hole could be to use something like Jeethu Rao's frozen module serializer and dump out all of the startup objects and put them into the BSS: https://bugs.python.org/issue34690 -- ___ Python tracker <https://bugs.python.org/issue40255> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39448] Add regen-frozen makefile target
Change by Neil Schemenauer : -- keywords: +patch pull_requests: +17558 pull_request: https://github.com/python/cpython/pull/18174 ___ Python tracker <https://bugs.python.org/issue39448> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39448] Add regen-frozen makefile target
New submission from Neil Schemenauer : Updating the frozen module "__hello__" code inside Python/frozen.c is currently a manual process. That's a bit tedious since it adds some extra work in the case that bytecode changes are made. I've created a small script and a makefile target to automates the process. -- messages: 360655 nosy: nascheme priority: normal severity: normal stage: patch review status: open title: Add regen-frozen makefile target type: enhancement ___ Python tracker <https://bugs.python.org/issue39448> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38006] Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit
Neil Schemenauer added the comment: New changeset 392a13bb9346331b087bcd8bb1b37072c126abee by Neil Schemenauer in branch 'master': bpo-38006: Add unit test for weakref clear bug (GH-16788) https://github.com/python/cpython/commit/392a13bb9346331b087bcd8bb1b37072c126abee -- ___ Python tracker <https://bugs.python.org/issue38006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38006] Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit
Neil Schemenauer added the comment: > It's fundamentally insane to expect any gc to work perfectly when it may be > blind to what the object graph _is_. I'm often amazed it works at all, let alone perfectly. ;-P This did trigger me to think of another possible problem. I setup my unit test as you suggested: # Z <- Y <- A--+--> WZ -> C # ^ | # +--+ # where: # WZ is a weakref to Z with callback C # Y doesn't implement tp_traverse # A contains a reference to itself, Y and WZ But what happens if the GC doesn't see that WZ is trash? Then it will not be cleared. Hang it off Y so the GC can't find it. We can set things up so that Z is freed before WZ (e.g. WZ in a second and newer cycle). Then, the callback might still run. On further thought, this seems safe (but maybe broken) because of the handle_weakrefs() logic. The GC will think WZ is going to outlive Z so it will call it before doing any tp_clear calls. -- ___ Python tracker <https://bugs.python.org/issue38006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38006] Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit
Change by Neil Schemenauer : -- pull_requests: +16348 pull_request: https://github.com/python/cpython/pull/16788 ___ Python tracker <https://bugs.python.org/issue38006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38006] Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit
Neil Schemenauer added the comment: Łukasz, is there some reason you removed old versions (2.7, 3.6, etc)? The bug is present on those versions of Python and it should be trivial to backport the fix. If those branches are maintained, I think we should fix it. Attached is a small test program (gc_weakref_bug_demo2.py) that causes the same crash as in this bug report (SEGV inside _PyFunction_Vectorcall). Setting things up is quite tricky and it depends on the order that things are cleared in delete_garbage(). Maybe still worth making a unit test for it? Instead of using the 'dummy' function, you can also use types.SimpleNamespace(). Calling repr() on it after tp_clear will result in a crash or an assert failure. Those two crashes are not needed to confirm the bug. Just the fact that 'callback' runs is proof enough. So, in a unit test, the callback to just set a global to indicate failure. -- Added file: https://bugs.python.org/file48637/gc_weakref_bug_demo2.py ___ Python tracker <https://bugs.python.org/issue38006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33418] Memory leaks in functions
Change by Neil Schemenauer : -- pull_requests: +16095 pull_request: https://github.com/python/cpython/pull/16502 ___ Python tracker <https://bugs.python.org/issue33418> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38006] Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit
Neil Schemenauer added the comment: I worked some on trying to create a unit test this evening. Attached is my best result so far (gc_weakref_bug_demo.py). It requires that you enable the 'xx' module so we have a container object without tp_traverse. We should probably add one to _testcapi for the real unit test so we have it. I can make the weakref callback run from within delete_garbage(). I haven't been able to make tp_clear for the function execute before the callback and I think that is needed to replicate the crash in this bug report. -- Added file: https://bugs.python.org/file48633/gc_weakref_bug_demo.py ___ Python tracker <https://bugs.python.org/issue38006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38006] Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit
Change by Neil Schemenauer : -- nosy: +benjamin.peterson, larry, ned.deily versions: +Python 2.7, Python 3.5, Python 3.6, Python 3.7 ___ Python tracker <https://bugs.python.org/issue38006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38006] Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit
Neil Schemenauer added the comment: I created GH-16502. I'm not exactly sure of the process of making a revert on a branch like '3.8' so hopefully it is correct. The code change is exactly what has been reverted in 3.8. The net effect will be as if the revert was never done. -- ___ Python tracker <https://bugs.python.org/issue38006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37725] "make clean" should remove PGO task data
Change by Neil Schemenauer : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue37725> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
