Eddie Elizondo added the comment:
It seems that we are back on track with perf being back to neutral!
I've created 4 new PRs. Each with an optimization applied on top of the
baseline introduction of instance immortalization.
The main PR 19474 currently stands at around 4%, after rebasing
Change by Eddie Elizondo :
--
pull_requests: +29621
pull_request: https://github.com/python/cpython/pull/31491
___
Python tracker
<https://bugs.python.org/issue40
Change by Eddie Elizondo :
--
pull_requests: +29620
pull_request: https://github.com/python/cpython/pull/31490
___
Python tracker
<https://bugs.python.org/issue40
Change by Eddie Elizondo :
--
pull_requests: +29619
pull_request: https://github.com/python/cpython/pull/31489
___
Python tracker
<https://bugs.python.org/issue40
Change by Eddie Elizondo :
--
pull_requests: +29618
pull_request: https://github.com/python/cpython/pull/31488
___
Python tracker
<https://bugs.python.org/issue40
Eddie Elizondo added the comment:
@eric.snow great to hear about this update! I'll start looking at some of the
techniques that we talked about to improve performance, I'm optimistic that
we'll be able to close down the gap to 2%.
--
___
Python
Eddie Elizondo added the comment:
> which would potentially save the two word per object overhead
Btw, I misread. I thought `gc_bits` referred to the bits used by the GC in the
reference count. In any case, you can still use a bit in the reference co
Eddie Elizondo added the comment:
> I'm somewhat puzzled how a version that does no more work and has no jumps is
> slower.
Oh I think I know why there's some confusion. I've updated the PR from the
initial version (which is probably the one that you saw). The branching does
les
Eddie Elizondo added the comment:
Mark:
> You are asking the whole world to take a hit on both performance and memory
> use.
That's an explicit non-goal. That's why the code was guarded to add as an
optional compilation mode
This should be added by default if and only if this is a n
Eddie Elizondo added the comment:
After the added optimizations and assuming that running this more rigorously
(i.e PGO + LTO + full cpu isolation + speed.python.org machine) also shows the
PR as perf neutral, what are people's thoughts so far?
Would this be enough to consider adding
Eddie Elizondo added the comment:
Neil:
> The fastest would be to create an immortal block as part of the BSS
> (uninitialized data).
That's an interesting idea, definitely worth exploring and we can probably get
some perf win out of it. And yes, using the frozen modules is defi
Eddie Elizondo added the comment:
I was able to get an equal (maybe slightly better) performance by following the
advice from Steve and Neil and skipping reference counting of instances that we
know are immortal.
It seems that by getting performance parity, we should be able to ease most
Eddie Elizondo added the comment:
> the CPU performance implications of adding a branch instruction to Py_INCREC
> and Py_DECREF were, unsurprisingly, quite high.
Yeah, makes sense. I guess it really depends on the specific profile of your
application.
For Instagram this was an overa
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +18829
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/19474
___
Python tracker
<https://bugs.python.org/issu
New submission from Eddie Elizondo :
Copy on writes are a big problem in large Python application that rely on
multiple processes sharing the same memory.
With the implementation of `gc.freeze`, we've attenuated the problem by
removing the CoW coming from the GC Head. However, reference
Eddie Elizondo added the comment:
> I'm concerned by release blocker because 3.9.0a3 version is supposed to be
> released soon, and usually release blocker do block a release :-)
Ah! That makes sense!
In any case, feel free to ping me if you need help on my side to get this PR
t
Eddie Elizondo added the comment:
The PR that I sent out already fixes the issue. @vstinner, @pablogsal, please
take a look again https://github.com/python/cpython/pull/18039
That should close this issue, no need to work around the bug priority
Eddie Elizondo added the comment:
Hey all, I've got a fix for this bug and the CI is green:
https://github.com/python/cpython/pull/18039
TL;DR: The module state have to be cleared at a later time. I explain in detail
in the PR.
Also, I didn't add a new test since there was a test
Change by Eddie Elizondo :
--
pull_requests: +17438
pull_request: https://github.com/python/cpython/pull/18039
___
Python tracker
<https://bugs.python.org/issue38
Eddie Elizondo added the comment:
PR with fix is out.
--
___
Python tracker
<https://bugs.python.org/issue35381>
___
___
Python-bugs-list mailing list
Unsub
Eddie Elizondo added the comment:
Victor, the PR with the fix is out. Easy catch after running it with my leak
detector!
--
___
Python tracker
<https://bugs.python.org/issue38
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +16856
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/17373
___
Python tracker
<https://bugs.python.org/issu
Eddie Elizondo added the comment:
Woops! I'll get to it before the end of the weekend!
--
___
Python tracker
<https://bugs.python.org/issue35381>
___
___
Pytho
Eddie Elizondo added the comment:
Woops! I'll get to it before the end of the weekend!
--
nosy: +eelizondo
___
Python tracker
<https://bugs.python.org/issue38
Eddie Elizondo added the comment:
Woops, just realized that you already documented this, thanks! Btw, Victor
already merged a fix for the windows compiler warning. This issue can be closed
now
--
___
Python tracker
<https://bugs.python.
Eddie Elizondo added the comment:
Hey Petr, I'll get to document this and fix the windows warning over the
weekend. I'll ping you on Github once it's ready
--
nosy: +eelizondo
___
Python tracker
<https://bugs.python.org/issue38
Eddie Elizondo added the comment:
The PR has been merged so the issue can be closed now
--
___
Python tracker
<https://bugs.python.org/issue38152>
___
___
Pytho
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +15738
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/16127
___
Python tracker
<https://bugs.python.org/issu
Eddie Elizondo added the comment:
I have a fix for this coming up.
--
nosy: +eelizondo
___
Python tracker
<https://bugs.python.org/issue38152>
___
___
Python-bug
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +15730
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/16115
___
Python tracker
<https://bugs.python.org/issu
Eddie Elizondo added the comment:
> Checking for refleak takes between 2 and 6 hours.
Ouch! Makes sense then.
We could potentially add a `pre-merge` job that only runs once the merge starts
to get executed. Anyways, that's a conversation for another time :-)
---
Thanks Stephane I'm v
Eddie Elizondo added the comment:
On it.
Also, I thought that the PR build bots already ran refleak tests by default? Do
you know why this it's not integrated to the PR flow?
--
nosy: +eelizondo
___
Python tracker
<https://bugs.python.
Change by Eddie Elizondo :
--
pull_requests: +15700
pull_request: https://github.com/python/cpython/pull/16078
___
Python tracker
<https://bugs.python.org/issue34
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +15697
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/16076
___
Python tracker
<https://bugs.python.org/issu
Change by Eddie Elizondo :
--
pull_requests: +15533
pull_request: https://github.com/python/cpython/pull/15892
___
Python tracker
<https://bugs.python.org/issue35
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +15040
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/15323
___
Python tracker
<https://bugs.python.org/issu
New submission from Eddie Elizondo :
`subtype_dealloc` is not correctly handling the reference count of c heap type
subclasses. It has some builtin assumptions which can result in the type
getting its reference count decreased more that it needs to be, leading to its
destruction when
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +12617
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue36531>
___
___
Py
New submission from Eddie Elizondo :
If a user accidentally defined more than one Py_tp_members in the spec,
PyType_FromSpec will ignore all but the last use case. However, the number of
members count will cause the type to allocate more memory than needed. This
leads to weird behavior
Eddie Elizondo added the comment:
> could we just remove the whole concept of heap allocated types?
I do have plans to start migrating more and more CPython modules to use heap
allocated types. For example I have posixmodule up in the queue (PR10854) and a
local change with all of
Eddie Elizondo added the comment:
Bump to get a review on the PR: https://github.com/python/cpython/pull/11661.
I believe all comments have now been addressed as well as adding a porting to
3.8 guide.
--
___
Python tracker
<ht
Eddie Elizondo added the comment:
> Please open a thread on python-dev
Done! https://mail.python.org/pipermail/python-dev/2019-February/156322.html
> Yes. You should add a new "Changes in the C API"
Done as well, I also included examples for the scenarios that will need
Eddie Elizondo added the comment:
Now, with that in mind - can you guys point me to the right thing to do now -
how can we move this forward? :)
* Should I write something up in python-dev/Discourse?
* Do I need to update the PY_VERSION_HEX?
* Do I need to write an entry in the Porting
Eddie Elizondo added the comment:
Thanks for the thorough feedback Stefan!
I would like to just add an extra thing to everything you already mentioned:
This change will NEVER cause a crash. The change that we are introducing is an
incref to a type object (no decrefs). Thus, there are two
Eddie Elizondo added the comment:
Hi Petr,
Please take a look at the Github PR. What do you think that's missing to move
this forward? I'd be more than happy to add more documentation/testing to it.
Let me know what you think
--
___
Python
Change by Eddie Elizondo :
--
keywords: +patch, patch, patch
pull_requests: +11461, 11462, 11463
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issu
Change by Eddie Elizondo :
--
keywords: +patch, patch
pull_requests: +11461, 11462
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issu
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +11461
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue35810>
___
___
Py
New submission from Eddie Elizondo :
Heap-allocated Types initializing instances through `PyObject_{,GC}_New{Var}`
will *NOT* not have their refcnt increased. This was totally fine under the
assumption that static types are immortal. However, heap-allocated types MUST
participate
Eddie Elizondo added the comment:
I also fixed the title to properly reflect what this is trying to achieve.
--
___
Python tracker
<https://bugs.python.org/issue35
Change by Eddie Elizondo :
--
title: Extension modules using non-API functions -> Cleanup extension functions
using _PyObject_LookupSpecial
___
Python tracker
<https://bugs.python.org/issu
Eddie Elizondo added the comment:
@vstinner: Sorry for not being clear - The intention of this change is two-fold:
1) Simplify the implementation of these functions.
2) Reduce the surface area of the C-API.
Given that the same functionality can be achieved with public functions of the
C-API
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +10316
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue35438>
___
___
Py
Eddie Elizondo added the comment:
Correction, this is not as trivial as just using `PyObject_GetAttrString`. Will
investigate the correct behavior.
--
___
Python tracker
<https://bugs.python.org/issue35
New submission from Eddie Elizondo :
Three extension modules: _testcapimodule.c, posixmodule.c, and mathmodule.c are
using `_PyObject_LookupSpecial` which is not API.
These should instead use `PyObject_GetAttrString`, `PyType_GetSlot`.
--
components: Library (Lib)
messages: 331364
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +10089
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue35381>
___
___
Py
Change by Eddie Elizondo :
--
title: Heap-allocated Posixmodule types -> Heap-allocated posixmodule types
___
Python tracker
<https://bugs.python.org/issu
Change by Eddie Elizondo :
--
title: Heap-allocated Posixmodule -> Heap-allocated Posixmodule types
___
Python tracker
<https://bugs.python.org/issu
New submission from Eddie Elizondo :
After bpo34784, there are still two more cases of statically allocated types
(DirEntryType & ScandirIteratorType). These should also be heap allocated to
make posixmodule fully compatible with PEP384.
--
components: Library (Lib)
messages: 33
Change by Eddie Elizondo :
--
pull_requests: +10032
___
Python tracker
<https://bugs.python.org/issue16086>
___
___
Python-bugs-list mailing list
Unsubscribe:
Change by Eddie Elizondo :
--
pull_requests: +9062
___
Python tracker
<https://bugs.python.org/issue34784>
___
___
Python-bugs-list mailing list
Unsubscribe:
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +8929
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue34784>
___
___
Py
New submission from Eddie Elizondo :
PyStructSequence_NewType does not currently work. Read the full analysis here:
https://mail.python.org/pipermail/python-dev/2018-September/155069.html
This aims to fix the implementation of PyStructSequence_NewType.
--
components: Library (Lib
Eddie Elizondo added the comment:
@ronaldoussoren
* This change currently works for all CPython. If you are using this pattern
then you probably want to be using PyType_FromSpec rather than having a static
PyTypeObject as discussed in PEP384: https://www.python.org/dev/peps/pep-0384
Eddie Elizondo added the comment:
@ronaldoussoren Please read the complete analysis from the mailing list:
https://mail.python.org/pipermail/python-dev/2018-August/154946.html. The
description here was just a rehash and I probably missed some context.
Particularly, when I said
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +8451
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue34533>
___
___
Py
New submission from Eddie Elizondo :
This applies the heap type refactoring from PEP 384 to the _csv module.
--
components: Extension Modules
messages: 324268
nosy: eelizondo
priority: normal
severity: normal
status: open
title: Apply PEP384 to _csv module
type: enhancement
versions
Change by Eddie Elizondo :
--
keywords: +patch
pull_requests: +8432
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue34522>
___
___
Py
New submission from Eddie Elizondo :
>From the documentation, it says that PyType_Ready should be called on `ALL`
>type objects to finish their initialization
>(https://docs.python.org/3/c-api/type.html#c.PyType_Ready). This means that a
>PyTypeObject's ob_type should a
Change by Eddie Elizondo <eduardo.elizondoru...@gmail.com>:
--
keywords: +patch
pull_requests: +5852
stage: -> patch review
___
Python tracker <rep...@bugs.python.org>
<https://bugs.pyt
Eddie Elizondo <eduardo.elizondoru...@gmail.com> added the comment:
@serhiy.storchaka tracemalloc can't distinguish between the usage of gc allocs,
normal mallocs, and free list reuse.
--
___
Python tracker <rep...@bugs.python.or
Eddie Elizondo <eduardo.elizondoru...@gmail.com> added the comment:
Currently, Python has very few instrumentation when it comes to the types of
allocations that it's doing. For instance, we currently can't identify how many
times an object uses free lists vs actual mallocs.
Cur
New submission from Eddie Elizondo <eduardo.elizondoru...@gmail.com>:
[WIP]
--
title: [WIP] Enhance Python's Memory Instrumentation with COUNT_ALLCOS ->
Enhance Python's Memory Instrumentation with COUNT_ALLOCS
___
Python tra
Change by Eddie Elizondo <eduardo.elizondoru...@gmail.com>:
--
nosy: elizondo93
priority: normal
severity: normal
status: open
title: Enhancing Python
type: enhancement
versions: Python 3.8
___
Python tracker <rep...@bugs.python.or
Change by Eddie Elizondo <eduardo.elizondoru...@gmail.com>:
--
title: [WIP] Enhancing Python's COUNT_ALLOCS -> [WIP] Enhance Python's Memory
Instrumentation with COUNT_ALLCOS
___
Python tracker <rep...@bugs.python.org>
<https
Change by Eddie Elizondo <eduardo.elizondoru...@gmail.com>:
--
title: Enhancing Python -> [WIP] Enhancing Python's COUNT_ALLOCS
___
Python tracker <rep...@bugs.python.org>
<https://bugs.pyt
New submission from Eddie Elizondo <eduardo.elizondoru...@gmail.com>:
The following build crashed:
mkdir debug && cd debug
../configure --with-pydebug
make EXTRA_CFLAGS="-DCOUNT_ALLOCS"
The bug was introduced here:
https://github.c
77 matches
Mail list logo
