Change by Neil Schemenauer :
--
pull_requests: +16092
pull_request: https://github.com/python/cpython/pull/16502
___
Python tracker
<https://bugs.python.org/issue38
Neil Schemenauer added the comment:
> Is introducing tp_clear on functions a thing that has ABI consequences? In
> other words, if we take our time on this, would it land in 3.8.1 or 3.9.0?
I think it should not have ABI consequences. However, I see the addition of
tp_clear as
Neil Schemenauer added the comment:
> Would [func tp_clear] help with memory usage in functions or was BPO-33418
> addressed in another way since?
Having a tp_clear for all container objects that can be involved in reference
cycles will help the GC free memory. BPO-33418 may be con
Neil Schemenauer added the comment:
Oops, I linked to wrong PR, my proposed fix is GH-16495. It is the same as
what Tim suggests in his last comment.
--
___
Python tracker
<https://bugs.python.org/issue38
Neil Schemenauer added the comment:
> Why setting it back to release blocker? As far as I recall, this issue is not
> a Python 3.8 regression. The regression which triggered this old and existing
> bug has been fixed, see previous comments.
I leave it up to our glorious release m
Change by Neil Schemenauer :
--
pull_requests: +16083
pull_request: https://github.com/python/cpython/pull/16495
___
Python tracker
<https://bugs.python.org/issue38
Neil Schemenauer added the comment:
> We can have finalizer code running during delete_garbage(). That
> code should not have access to non-valid objects. Weakrefs seem be
> a way to violate that. handle_weakrefs() take care of some of them
> but it seems there are other issues.
Neil Schemenauer added the comment:
> Fleshing out something I left implicit: if there's a trash object T
> with a finalizer but we don't KNOW it's trash, we won't force-run its
> finalizer before delete_garbage starts either. Then, really the same
> thing: we
Neil Schemenauer added the comment:
Since W is in the unreachable set, we should not be executing its callback.
Would the attached rough patch (gc_disable_wr_callback.txt) be a possible fix?
When we find W inside handle_weakrefs(), we mark it as trash and will not
execute the callback
Neil Schemenauer added the comment:
I hacked up my copy of CPython to add flags to PyObject to see the
GC state of objects. That makes it easier to know if an object is
in the 'unreachable' list or not.
> We must know that F is trash, else we never would have called tp_clear(
Neil Schemenauer added the comment:
Victor:
> I'm surprised that the function would be seen as "unreachable" if
> it's reference counter was equal to 135:
If all those references are contained within that garbage set, it is
possible.
Pablo:
> For the weakre
Neil Schemenauer added the comment:
I think the problem with your logic is that the weakref to F is part of the
garbage set. So, handle_finalizers() should detect that and clear the
finalizer rather than call it. Once we get to delete_garbage() and start
calling tp_clear(), we can'
Neil Schemenauer added the comment:
A few comments from the "peanut gallery". Thanks to Victor and Pablo for doing
the hard work of investigating this bug.
First, it has been a long time since Tim and I first developed gcmodule.c. So,
some of my understanding may be inaccurate d
Change by Neil Schemenauer :
--
keywords: +patch
pull_requests: +14790
pull_request: https://github.com/python/cpython/pull/15033
___
Python tracker
<https://bugs.python.org/issue37
New submission from Neil Schemenauer :
I find it annoying and surprising that "make clean" does not remove the PGO
data. If you change a source file, running "make clean" and "make" should
result in a newly built executable, IMHO. As it is now, you usually
Neil Schemenauer added the comment:
New changeset 382cb85401bb010ead411c0532499ffe16c3cf27 by Neil Schemenauer
(Miss Islington (bot)) in branch '3.8':
bpo-37707: Exclude expensive unit tests from PGO task (GH-15009) (#15024)
https://github.com/python/cpyt
Change by Neil Schemenauer :
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Neil Schemenauer added the comment:
Closing as I think PR 14702 mostly resolves this. The gc_collect() issue has
not been resolved but based on some investigation, I think it is not worth the
effort to try to reduce the time spend in that function. Reducing the number
of unit tests being
Neil Schemenauer added the comment:
New changeset 52a48e62c6a94577152f9301bbe5f3bc806cfcf1 by Neil Schemenauer in
branch 'master':
bpo-37707: Exclude expensive unit tests from PGO task (GH-15009)
https://github.com/python/cpython/commit/52a48e62c6a94577152f9301bbe5f3
Neil Schemenauer added the comment:
I think expanding the list of tests used by --pgo is fine, as long as we put a
little thought into each one we add. The ones added by Steve look fine to me.
It seems useful to run a profiler and see if there are any unusually expensive
tests being added
Change by Neil Schemenauer :
--
keywords: +patch
pull_requests: +14772
pull_request: https://github.com/python/cpython/pull/15009
___
Python tracker
<https://bugs.python.org/issue37
New submission from Neil Schemenauer :
Add a new support decorator, @skip_if_pgo_task and then use it to mark test
cases. I suspect the PGO task works well if it can exercise common code paths.
Running long tests likely have rapidly diminishing benefits. The instrumented
PGO executable
Neil Schemenauer added the comment:
New changeset 4e16a4a3112161a5c6981c0588142d4a4673a934 by Neil Schemenauer in
branch 'master':
bpo-36044: Reduce number of unit tests run for PGO build (GH-14702)
https://github.com/python/cpython/commit/4e16a4a3112161a5c6981c0588142d
Neil Schemenauer added the comment:
I changed configure.in to use AC_ARG_VAR instead. So, you can override it as
you suggest:
./configure [..] PROFILE_TASK="-m test --pgo-extended"
--
___
Python tracker
<https://bugs.python.o
Neil Schemenauer added the comment:
Thanks for the feedback. I agree that putting the tests in regrtest is better.
I've made the following changes:
- the --pgo option now uses the shorter list of tests by default
- I added --pgo-extended as a regrtest option that enables the old beh
Neil Schemenauer added the comment:
+-+---+-+
| Benchmark | task-all2 | task-short7 |
+=+===+=+
| 2to3| 304 ms| 305 ms
Neil Schemenauer added the comment:
I tweaked the list of unit tests a little more, trying to incorporate some from
your Docker build settings. Not sure what's going on with the pickle results.
Below are new pyperformance runs, comparing my PR to the "master" version it i
Change by Neil Schemenauer :
--
keywords: +patch
pull_requests: +14501
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/14702
___
Python tracker
<https://bugs.python.org/issu
Neil Schemenauer added the comment:
> Decreasing the total wall time for a default --enable-optimizations build
> would
> be a good thing for everyone, provided the resulting interpreter remains
> "effectively similar" in speed. If you somehow manage to find som
Neil Schemenauer added the comment:
New changeset 5d25f2b70351fc6a56ce5513ccf5f58556c18837 by Neil Schemenauer in
branch 'master':
bpo-37537: Compute allocated blocks in _Py_GetAllocatedBlocks() (#14680)
https://github.com/python/cpython/commit/5d25f2b70351fc6a56ce5513ccf5f5
Neil Schemenauer added the comment:
PyPerformance result, using --enable-optimizations
+-+-+--+
| Benchmark | base| no-allocated
Change by Neil Schemenauer :
--
keywords: +patch
pull_requests: +14486
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/14680
___
Python tracker
<https://bugs.python.org/issu
New submission from Neil Schemenauer :
This is a small but measurable optimization for _PyObject_Malloc and
_PyObject_Free. It slows down _Py_GetAllocatedBlocks a bit but I think that's
a price worth paying.
--
components: Interpreter Core
messages: 347602
nosy: nascheme
pri
Change by Neil Schemenauer :
--
keywords: +patch
pull_requests: +14291
pull_request: https://github.com/python/cpython/pull/14474
___
Python tracker
<https://bugs.python.org/issue37
Change by Neil Schemenauer :
Added file: https://bugs.python.org/file48446/obmalloc_overhead.py
___
Python tracker
<https://bugs.python.org/issue37448>
___
___
Python-bug
Change by Neil Schemenauer :
Added file: https://bugs.python.org/file48445/pyperf_radix_compare.txt
___
Python tracker
<https://bugs.python.org/issue37448>
___
___
Pytho
New submission from Neil Schemenauer :
This patch implements an alternative version of obmalloc's address_in_range().
It uses a radix tree to map the areas of memory covered by obmalloc arenas.
pymalloc_free() uses address_in_range() to determine if a block of memory is
controll
Neil Schemenauer added the comment:
I suggest we change PyCode_New() back in the next beta, despite that change
breaking Cython again. We could work with them so that they have a new release
with a "PY_VERSION_HEX > 0x030800b1" test in it. Try to get that Cython version
re
Neil Schemenauer added the comment:
Updated patch is attached. It appears that the extra item is only needed if
Py_TPFLAGS_TYPE_SUBCLASS set. In all other cases, it seems we don't need the
extra space for the sentinel. At least, the unit tests pass with this change.
It could be
New submission from Neil Schemenauer :
In the process of working on some garbage collector/obmalloc experiments, I
noticed what seems to be a quirk about PyType_GenericAlloc(). It calls:
size = _PyObject_VAR_SIZE(type, nitems+1);
Note the "+1" which is documented as "f
Neil Schemenauer added the comment:
We now have a concrete use case. ;-)
My idea was that we can introduce a new, CPython internal API that
aligns on 8-byte boundaries (or takes alignment as a parameter). The
API would be a stop-gap measure. We can use the API to reduce
the overhead for
Neil Schemenauer added the comment:
> sys.getsizeof(3.14) is 24. And it becomes 32 byte in 16byte aligned
> pymalloc. (+33%)
I've been doing some reading and trying to understand this issue. My
understanding is that malloc() needs to return pointers that are 16-byte
aligned o
Neil Schemenauer added the comment:
I think there are two questions to answer. First, do we want to support
multiple runtimes per process? Second, if we do, what is the best way to do
that? Some people would argue that multiple runtimes are not needed or are too
hard to do. Maybe they
Neil Schemenauer added the comment:
Sorry, morning coffee didn't kick in yet I guess. ;-) My actual wish is to
make all types heap allocated and eliminate the statically allocated ones. So,
Py_TPFLAGS_HEAPTYPE would be set on all types in that world. That is a
gigantic task, affe
Neil Schemenauer added the comment:
Hello Eddie,
Thank you for putting what looks to be significant effort into this PR. It
would be great if we can get this fixed. There is a real issue about breaking
3rd party extensions. So, we want to proceed with care.
I wonder, if we are going to
New submission from Neil Schemenauer :
I was doing some 'perf' profiling runs of Python. I decided to try running
PROFILE_TASK to see what the profile looks like. I was surprised that
gc_collect dominated runtime:
Children Self Symbol
+ 93.93
Neil Schemenauer added the comment:
BTW, 'perf report [...]' has a really neat annotated assembly view. Scroll to
the function you are interested in and press 'a'. Press 't' to toggle the time
units (left side numbers). I'm attaching a screenshot
Neil Schemenauer added the comment:
Some profiling using 'perf'. This is for cpython
63fa1cfece4912110ce3a0ff11fb3ade3ff5e756.
children self
[...]
+ 97.27% 0.00% run_mod (inlined)
+ 88.53% 6.33% PyObject_SetAttr
+ 79.34% 6.80% type_setattro
+ 43.92
Neil Schemenauer added the comment:
Some thoughts on this API.
I think we need the "create with exclusive behavior" option, even though we
don't know how to implement it on Windows right now. To me, there are two
cases when calling SharedMemory:
1) You want to create a ne
Change by Neil Schemenauer :
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Neil Schemenauer added the comment:
New changeset 5741c45acf9b0ce22ff0dbf56322fe0ff16cfcfc by Neil Schemenauer in
branch 'master':
bpo-35903: Use autoconfig to probe for shm_open() and shm_unlink(). (#11765)
https://github.com/python/cpython/commit/5741c45acf9b0ce22ff0dbf56322fe
Neil Schemenauer added the comment:
I didn't finish reviewing completely yet but here are some comments.
I think the random filename generation can be pulled out of
_posixshmem. Make it require that a filename is passed into it.
That moves a fair bit of complexity out of C and into P
Neil Schemenauer added the comment:
Hi Ronald,
I tested with my MacBook but I'm not sure what version of the OS. Probably the
latest but I can check when I get home. Maybe my comment was not clear. On my
MacBook, you don't need librt for shm_open, etc. That sounds like it is th
Neil Schemenauer added the comment:
I didn't understand autoconfig well enough but I think I fixed it now.
Apparently MacOS is a platform where it has shm_open but you don't need to link
with -lrt. The revised 'configure' should handl
Change by Neil Schemenauer :
--
pull_requests: +11719, 11720
___
Python tracker
<https://bugs.python.org/issue35903>
___
___
Python-bugs-list mailing list
Unsub
Change by Neil Schemenauer :
--
pull_requests: +11719, 11720, 11721
___
Python tracker
<https://bugs.python.org/issue35903>
___
___
Python-bugs-list mailin
Change by Neil Schemenauer :
--
pull_requests: +11719
___
Python tracker
<https://bugs.python.org/issue35903>
___
___
Python-bugs-list mailing list
Unsubscribe:
New submission from Neil Schemenauer :
The logic in setup.py that determines if _multiprocessing/posixshmem.c should
get built is not very robust. I think it is better to use autoconfig to probe
for the required functions and libraries. My autoconfig brain cells are a bit
fuzzy but I think
Change by Neil Schemenauer :
--
nosy: +nascheme
___
Python tracker
<https://bugs.python.org/issue23903>
___
___
Python-bugs-list mailing list
Unsubscribe:
Neil Schemenauer added the comment:
I too would prefer a new method name rather than overloading splitlines() with
more keyword args (passed as hardcoded constants, usually). Again, I think we
want:
list(open(..).read().()) == list(open(..))
readlines() returns a list but I think this
Neil Schemenauer added the comment:
> Why not simply add a new parameter, to make people who want ASCII linebreaks
> continue to use .splitlines() ?
That could work but I think in nearly every case you don't want to use
splitlines() without supplying the parameter. So, it seems
Neil Schemenauer added the comment:
I've created a topic on this inside the "Ideas" area of discuss.python.org.
Sorry if that wasn't appropriate, not sure if I should have keep the discussion
here.
Inada Naoki suggests creating a new method str.iterlines{[keepends]).
Neil Schemenauer added the comment:
If we introduce a keyword parameter, I think the default of str.splitlines()
should be changed to match bytes.splitlines (and match Python 2
str.splitlines()). I.e. split on \r and \n by default. I looked through the
stdline and I can't find any
Neil Schemenauer added the comment:
I just found bug #22232 myself but thanks for pointing it out.
> changing the behavior unconditionally is not an option
At this point, I disagree. If I do a search on the web, lots of pages
referring to str.splitlines() seem it imply that is splits o
Neil Schemenauer added the comment:
New patch that changes str.splitlines to work like Python 2 str.splitlines and
like Python 3 bytes.splitlines. Surprisingly, only a few cases in the unit
test suite fail. I've fixed them in my patch.
--
Added file: https://bugs.pytho
Neil Schemenauer added the comment:
Some further progress on this. My patch slows down reading files with the
codecs module very significantly. So, I think it could never be merged as is.
Maybe we would need to implement an alternative str.splitlines that behaves as
we want, implemented
Neil Schemenauer added the comment:
Attached is a rough patch that tries to fix this problem. I changed the
behavior in that unicode char 0x2028 is no longer treated as a line separator.
It would be trival to change the regex to support that too, if we want to
preserve backwards
Neil Schemenauer added the comment:
I think one bug here is that codecs readers use str.splitlines() internally.
The splitlines method treats a bunch of different characters as line
separators, unlike io..readlines(). So, you end up with different
behavior between doing iter
Neil Schemenauer added the comment:
There is another issue related to this. If you use codecs to get a reader, it
uses str.splitlines() internally, which treats a bunch of different characters
as line terminators. See issue #18291 and:
https://docs.python.org/3.8/library/stdtypes.html
Neil Schemenauer added the comment:
Perhaps the 'csv' module should do some sanity checking on the file passed to
the reader. The docs recommend that newline='' be used to open the file.
Maybe 'csv' could check that and warn if its not the case. I poked arou
Neil Schemenauer added the comment:
Thank you for the research. The problem is indeed that \v is getting treated as
a line separator. That is an intentional design choice, see:
https://bugs.python.org/issue12855
It would seem to have some surprising implications for CSV parsing. E.g. if
Neil Schemenauer added the comment:
> The problem with a SyntaxWarning is that the wrong people will see it. It
> gets in the way of users of applications that happen to be written in Python.
Turn the check on only when PYTHONDEVMODE is set? Seems like it solves the
issue with the
Neil Schemenauer added the comment:
Woudn't turning these off hurt performance a lot? If so, I don't know if
people would actually use such a mode. Then it becomes pretty useless. Could
we combine this idea with the PYTHONDEVMODE flag? If PYTHONDEVMODE is turned
on, we could
Neil Schemenauer added the comment:
Hello Serhiy,
I've not reviewed the patch but I trust that if you say it simplifies things,
that's so. Perhaps the important question is if it is okay to change the AST
in backwards incompatible ways within 3.x releases. As a library author
Neil Schemenauer added the comment:
Assuming it is not crazy complicated to fix, I would like to to be changed. It
should work like the TypeError example. That fact that it has worked the
current way since Python 1.5 isn't a strong argument. I think no one should be
surprised
New submission from Neil Schemenauer :
This seems to be a bug in codecs.getreader(). io.TextIOWrapper(fp, encoding)
works correctly.
--
files: codecs_bug.py
messages: 326382
nosy: nascheme
priority: low
severity: normal
status: open
title: codecs.getreader() splits lines containing
Neil Schemenauer added the comment:
I've checked Quixote PTL module support. I will have to make some changes to
support the removal of Str and other node types. E.g. I have to change
visit_Str to visit_Constant. The fixes are pretty easy though and I think it
is reasonable that
Neil Schemenauer added the comment:
As someone who does AST manipulation (Quixote PTL template support), I'm
interested in this patch. I think what is proposed sounds like a good change.
I.e. having the extra node types is not useful and makes the compiler harder to
understand. Prov
Change by Neil Schemenauer :
--
pull_requests: +8767
___
Python tracker
<https://bugs.python.org/issue34704>
___
___
Python-bugs-list mailing list
Unsubscribe:
New submission from Neil Schemenauer :
My long term goal is to make it possible to make PyObject an opaque structure.
Obviously that would break nearly every 3rd party extension at this point.
However, to allow freedom on certain interpreter implementation strategies, it
is helpful if code
Change by Neil Schemenauer :
--
nosy: +nascheme
___
Python tracker
<https://bugs.python.org/issue26979>
___
___
Python-bugs-list mailing list
Unsubscribe:
Neil Schemenauer added the comment:
I would suggest that sharing of objects between interpreters should be stamped
out. Could we have some #ifdef debug checking that would warn or assert so
this doesn't happen? I know currently we share a lot of objects. However, in
the long term,
Neil Schemenauer added the comment:
I commented on python-dev but maybe it is better to keep discussion here.
Could we make the frozenmodules thing into a dynamically loaded module? Then
you could have support for end users making their own. E.g. a command-line
param that lists a set of
Neil Schemenauer added the comment:
Yes, it looks like the same issue as bpo-29512. Renaming test.bisect is the
simplest solution. I have trained myself to run "python -m test.regrtest " so this issue doesn't affect me any more. However, I think it was a
trap that will ca
Neil Schemenauer added the comment:
This is still broken, IMHO. Either we should rename test.bisect or we should
remove all of the 'if __name__ == "__main__"' part of the test scripts. You
can't run the tests reliably as scripts anymore. Doing so puts Lib/t
Neil Schemenauer added the comment:
Closing this as the current makefile works much better in this regard, at least
based on my experience.
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracke
Change by Neil Schemenauer :
--
keywords: +patch
pull_requests: +7804
___
Python tracker
<https://bugs.python.org/issue34106>
___
___
Python-bugs-list mailin
New submission from Neil Schemenauer :
This patch is based on top of GH-8229 but could be changed to work without it.
--
components: Build
messages: 321580
nosy: nascheme
priority: normal
severity: normal
stage: patch review
status: open
title: Add --with-module-config= to 'conf
Neil Schemenauer added the comment:
I removed the "makesetup_fallback_dist.txt". PR-8260 is a more polished
implementation of the same idea.
--
___
Python tracker
<https://bugs.python.o
Change by Neil Schemenauer :
Removed file: https://bugs.python.org/file47685/makesetup_fallback_dist.txt
___
Python tracker
<https://bugs.python.org/issue32430>
___
___
Change by Neil Schemenauer :
--
pull_requests: +7793
___
Python tracker
<https://bugs.python.org/issue32430>
___
___
Python-bugs-list mailing list
Unsubscribe:
Change by Neil Schemenauer :
Added file: https://bugs.python.org/file47685/makesetup_fallback_dist.txt
___
Python tracker
<https://bugs.python.org/issue32430>
___
___
Neil Schemenauer added the comment:
PR 8229 doesn't work if the build dir is separate from the src dir (mentioned
in the PR comments). The attached patch (makesetup_fallback_dist.txt) changes
makesetup to fallback to using $srcdir/Modules/Setup.dist if Modules/Setup
doesn't
Change by Neil Schemenauer :
--
keywords: +patch
pull_requests: +7770
___
Python tracker
<https://bugs.python.org/issue34089>
___
___
Python-bugs-list mailin
New submission from Neil Schemenauer :
This is related to Issue32430. The behavior of Modules/Setup.dist and
Modules/Setup is quirky and problematic. Issue32430 links to some of the
related issues. The problem discussed in Issue24575 is also related
(Py_BUILD_CORE not being defined when
Neil Schemenauer added the comment:
I ran into an error like this and perhaps I have a clue was why some people run
into it. Dynamic modules built from Modules/Setup will need to have
-DPy_BUILD_CORE defined within the Setup file in the case they need to use the
_Py_BEGIN_SUPPRESS_IPH
Neil Schemenauer added the comment:
New changeset d3ed67d14ed401dfe2b5d07b6941adc3ecacb268 by Neil Schemenauer in
branch 'master':
bpo-33609: small wording fixes to dict ordering docs
https://github.com/python/cpython/commit/d3ed67d14ed401dfe2b5d07b6941ad
Change by Neil Schemenauer :
--
pull_requests: +7110
___
Python tracker
<https://bugs.python.org/issue33609>
___
___
Python-bugs-list mailing list
Unsubscribe:
Neil Schemenauer added the comment:
The wording sounds strange to me. Currently it is:
> Note that updating key doesn't affects the order ...
I would say:
> Note that updating a key does not affect the order ...
--
nosy: +nascheme
___
Change by Neil Schemenauer :
--
keywords: +patch
pull_requests: +5941
___
Python tracker
<https://bugs.python.org/issue33124>
___
___
Python-bugs-list mailin
101 - 200 of 375 matches
Mail list logo