[Python-Dev] Memory bitmaps for the Python cyclic garbage collector

Python objects that participate in cyclic GC (things like lists, dicts, sets but not strings, ints and floats) have extra memory overhead. I think it is possible to mostly eliminate this overhead. Also, while the GC is running, this GC state is mutated, which destroys copy-on-write optimizations.

Re: [Python-Dev] Consolidate stateful runtime globals

Is there any issue with unit-at-a-time optimization? I would imagine that a static global would allow optimizations that are not safe for a exported global (not sure the C term for it). I suspect it doesn't matter and I support the idea in general. Global variables in extension modules kills the

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

Larry Hastings wrote: > The TL;DR summary: add support for property objects to modules. > I've already posted a prototype. I posted an idea to python-ideas about lazy module loading. If the lazy loading idea works, having properties would allow modules to continue to be "lazy safe" but to easily

[Python-Dev] Lazy initialization of module global state

This is an idea that came out of the lazy module loading (via AST analysis), posted to python-ideas. The essential idea is to split the marshal data stored in the .pyc into smaller pieces and only load the parts as they are accessed. E.g. use a __getattr__ hook on the module to unmarshal+exec the

Re: [Python-Dev] Python startup optimization: script vs. service

Christian Heimes wrote: > That approach could work, but I think that it is the wrong > approach. I'd rather keep Python optimized for long-running > processes and introduce a new mode / option to optimize for > short-running scripts. Another idea is to run a fake transasaction through the process

Re: [Python-Dev] Reorganize Python categories (Core, Library, ...)?

Antoine Pitrou wrote: > There is no definite "correct category" when you're mixing different > classification schemes (what kind of bug it is -- > bug/security/enhancement/etc. --, what functional domain it pertains > to -- networking/concurrency/etc. --, which stdlib API it affects). I think the

[Python-Dev] Enabling depreciation warnings feature code cutoff

On 2017-11-06, Nick Coghlan wrote: > Gah, seven years on from Python 2.7's release, I still get caught by > that. I'm tempted to propose we reverse that decision and go back to > enabling them by default :P Either enable them by default or make them really easy to enable for development evironment

Re: [Python-Dev] 'continue'/'break'/'return' inside 'finally' clause

Serhiy Storchaka wrote: > Currently 'break' and 'return' are never used inside 'finally' > clause in the stdlib. See the _recv_bytes() function: Lib/multiprocessing/connection.py: 316 > I would want to see a third-party code that uses them. These are the only ones I found so far: ./gevent/src

Re: [Python-Dev] 'continue'/'break'/'return' inside 'finally' clause

Generally I think programming language implementers don't get to decide how the language works. You just have to implement it as specified, inconvenient as that might be. However, from a languge design prespective, I think there is a good argument that this is a corner of the language we should co

Re: [Python-Dev] 'continue'/'break'/'return' inside 'finally' clause

On 2018-01-03, Serhiy Storchaka wrote: > I haven't found 'finally' clauses in > https://github.com/gevent/gevent/blob/master/src/gevent/libev/corecffi.py. > Perhaps this code was changed in recent versions. Yes, I was looking at was git revision bcf4f65e. I reran my AST checker and found this: .

Re: [Python-Dev] 'continue'/'break'/'return' inside 'finally' clause

On 2018-01-03, Guido van Rossum wrote: > I'm sorry, I don't think more research can convince me either way. > I want all three of return/break/continue to work inside finally > clauses, despite there being few use cases. That's fine. The history of 'continue' inside 'finally' is interesting. The

Re: [Python-Dev] 'continue'/'break'/'return' inside 'finally' clause

On 2018-01-04, Guido van Rossum wrote: > We should interview you for the paper we may be writing for HOPL. History of Programming Languages? I did some more digging this afternoon, trying to find source code between versions 1.0.1 and 0.9.1. No luck though. It looks like 0.9.1 might have been t

Re: [Python-Dev] Python startup time

Antoine: > The overhead of importing is not in trying too many names, but in > loading the module and executing its bytecode. That was my conclusion as well when I did some profiling last fall at the Python core sprint. My lazy execution experiments are an attempt to solve this: https://gith

Re: [Python-Dev] Python startup time

On 2018-05-03, Lukasz Langa wrote: > > On May 2, 2018, at 8:57 PM, INADA Naoki wrote: > > * Add lazy compiling API or flag in `re` module. The pattern is compiled > > when first used. > > How about go the other way and allow compiling at Python > *compile*-time? That would actually make things f

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

On 2018-05-25, Antoine Pitrou wrote: > Do you have something specific in mind? I think compressed by default is a good idea. My quick proposal: - Use fast compression like lz4 or zlib with Z_BEST_SPEED - Add a 'compress' keyword argument with a default of None. For protocol 5, None means to

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

On 2018-05-25, Antoine Pitrou wrote: > The question is what purpose does it serve for pickle to do it rather > than for the user to compress the pickle themselves. You're basically > saving one line of code. It's one line of code everywhere pickling or unpicking happens. And you probably need to

Re: [Python-Dev] When tp_clear returns non-zero?

On 2018-05-28, Serhiy Storchaka wrote: > I'm interesting what the result of this function means. In what > cases it can return non-zero, and can it set an exception? My memory is fuzzy (nearly 20 years since I wrote that code). My best guess is that I thought a return value might be useful someho

[Python-Dev] obmalloc mmap/munmap thrashing

I was running Python 2.4.11 under strace and I noticed some odd looking system calls: mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9848681000 munmap(0x7f9848681000, 262144) = 0 mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)

Re: [Python-Dev] PyGC_Collect ignores state of `enabled`

Hi, I intended for gc.collect() to actually collect cycles even if the auto-GC was disabled. Having Py_Finalize() run GC even when it has been disabled seems wrong to me. Originally, cyclic GC was supposed to be optional. Back then, most programs did not leak cycles. I would vote for Py_Finali

Re: [Python-Dev] PyGC_Collect ignores state of `enabled`

Benjamin Peterson wrote: > Adding PyGC_CollectIfEnabled() and calling it in Py_Finalize is probably > fine. I don't think the contract of PyGC_Collect itself (or gc.collect() > for that matter) should be changed. You might want to disable GC but > invoke it yourself. Yes, that sounds okay to me.

Re: [Python-Dev] PyGC_Collect ignores state of `enabled`

Nick Coghlan wrote: > PEP 3121 is insufficient, since a lot of extension modules can't (or > at least haven't) adopted it in practice. > https://www.python.org/dev/peps/pep-0489/ has some more background on > that (since it was the first step towards tackling the problem in a > different way that

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

This looks pretty good to me. I don't think we should limit operands based on type, that's anti-Pythonic IMHO. We should use duck-typing and that means a special method, I think. We could introduce a new one but __bytes__ looks like it can work. Otherwise, maybe __ascii__ is a good name. Object

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

Neil Schemenauer wrote: > We should use duck-typing and that means a special method, I > think. We could introduce a new one but __bytes__ looks like it > can work. Otherwise, maybe __ascii__ is a good name. I poked around the Python 3 source. Using __bytes__ has some downsides,

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

Antoine Pitrou wrote: > On Wed, 15 Jan 2014 15:47:43 + (UTC) Neil S wrote: >> >> Objects that implement __str__ can also implement __bytes__ if they >> can guarantee that ASCII characters are always returned, no matter >> what the *value* > > I think that's a slippery slope. __bytes__ should

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

Glenn Linderman wrote: > On 1/15/2014 7:52 AM, Eric V. Smith wrote: >> Either that, or we're back to encoding the result of __format__ and >> accepting that sometimes it might throw errors, depending on the values >> being passed into format(). That would take us back to Python 2 hell. Please no

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

Greg Ewing wrote: > Neil Schemenauer wrote: >> Objects that implement __str__ can also implement __bytes__ if they >> can guarantee that ASCII characters are always returned, > > I think __ascii_ would be a better name. I'd expect > a method called __bytes__ on an in

Re: [Python-Dev] PEP 461 updates

Carl Meyer wrote: > I think the PEP could really use a rationale section summarizing _why_ > these formatting operations are being added to bytes I agree. My attempt at re-writing the PEP is below. >> In order to avoid the problems of auto-conversion and >> value-generated exceptions, all objec

Re: [Python-Dev] PEP 461 - Adding % and {} formatting to bytes

Michael Urman wrote: > If __format__ receives a format_spec of type bytes, it should return > bytes. For such cases on objects that cannot support bytes (i.e. for > str), it can raise. This appears to avoid the need for additional > methods. (As does Nick's proposal of leaving it out for now.) Th

Re: [Python-Dev] PEP 461 updates

Greg wrote: > I don't think it matters whether the internal details of that > debate make sense to the rest of us. The main thing is that > a consensus seems to have been reached on bytes formatting > being basically a good thing. I've been mostly steering clear of the metaphysical and writing co

[Python-Dev] Migration from Python 2.7 and bytes formatting

As I see it, there are two separate goals in adding formatting methods to bytes. One is to make it easier to write new programs that manipulate byte data. Another is to make it easier to upgrade Python 2.x programs to Python 3.x. Here is an idea to better address these separate goals. Introduce

Re: [Python-Dev] PEP 461 Final?

Ethan Furman wrote: > Overriding Principles >= > > In order to avoid the problems of auto-conversion and Unicode exceptions that > could plague Py2 code, all object checking will be done by duck-typing, not by > values contained in a Unicode representation [3]_. I think a long

Re: [Python-Dev] PEP 461 Final?

Mark Lawrence wrote: > Using %b could cause problems in the future as b is used in new style > formatting to mean output numbers in binary, so %B seems to me the > obvious choice as it's also unused. After updating my patch, I've decided that %s works better. My patch implements PEP 461 as pro

Re: [Python-Dev] Migration from Python 2.7 and bytes formatting

I've refined this idea a little in my latest PEP 461 patch (issue 20284). Continuing to use %s instead of introducing %b seems better. I've called the commmand-line option -2, it could be used to enable other similar porting aids. I'd like to try porting code making use of the -2 feature to see

Re: [Python-Dev] Migration from Python 2.7 and bytes formatting

On 2014-01-17, Ryan Gonzalez wrote: > A command line parameter?? I believe it has to be global flag. A __future__ statement will not work. Probably we should allow the flag to be set with an environment variable as well. > The annoying part would be telling every single user to call Python with

Re: [Python-Dev] Migration from Python 2.7 and bytes formatting

Glenn Linderman wrote: > -1 overall. > > Not worth the extra complexity in documentation and command line > parameters. Really? It's less than 20 lines of code to implement, probably similar to document. With millions maybe billions of lines of existing Python 2.x code to port, I'm dismayed to

Re: [Python-Dev] Migration from Python 2.7 and bytes formatting

On 2014-01-18, Stephen J. Turnbull wrote: > The above are descriptions of current behavior (ie, unchanged by PEPs > 460, 461), and this: [..] > is the content of this proposal, is that right? The proposal is that -2 enables the following: - %r as an alias for %a (i.e. calls ascii()) - %s wil

Re: [Python-Dev] PEP 461 Final?

Ethan Furman wrote: > So, if %a is added it would act like: > > - >"%a" % some_obj > - >tmp = str(some_obj) >res = b'' >for ch in tmp: >if ord(ch) < 256: >res += bytes([ord(ch)] >else: >res += unicode_escape(ch) > - >

Re: [Python-Dev] PEP 461 Final?

Steven D'Aprano wrote: >> To properly handle int and float subclasses, int(), index(), and float() >> will be called on the objects intended for (d, i, u), (b, o, x, X), and >> (e, E, f, F, g, G). > > > -1 on this idea. > > This is a rather large violation of the principle of least surprise, and

Re: [Python-Dev] Let's change to C API!

On 2018-07-31, Victor Stinner wrote: > I would be nice to be able to use something to "generate" C > extensions, maybe even from pure Python code. But someone has to > work on a full solution to implement that. Perhaps a "argument clinic on steroids" would be the proper approach. So, extensions w

Re: [Python-Dev] bpo-34595: How to format a type name?

On 2018-09-13, Victor Stinner wrote: > Right, that's a side effect of the discussion on the C API. It seems > like Py_TYPE() has to go in the new C API. Sorry, the rationale is not > written down yet, but Dino convinced me that Py_TYPE() has to go :-) My understanding is that using Py_TYPE() insid

Re: [Python-Dev] Heap-allocated StructSequences

On 2018-09-04, Eddie Elizondo wrote: > Solution: > > * Fix the implementation of PyStructSequence_NewType: > > The best solution would be to fix the implementation of this > function. This can easily be done by dynamically creating a > PyType_Spec and calling PyType_FromSpec Hello Eddie, Th

Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?

On 2018-09-14, Larry Hastings wrote: [...] > improvement 0.21242667903482038 % I assume that should be 21.2 % othewise I recommend you abandon the idea. ;-P > The downside of the patch: for these modules it ignores the Python files on > disk--it doesn't even stat them. Having a command-line/env

Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?

On 2018-09-14, Larry Hastings wrote: > [..] adding the stat calls back in costs you half the startup.  So > any mechanism where we're talking to the disk _at all_ simply > isn't going to be as fast. Okay, so if we use hundreds of small .pyc files scattered all over the disk, that's bad? Who would

Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?

On 2018-09-15, Paul Moore wrote: > On Fri, 14 Sep 2018 at 23:28, Neil Schemenauer wrote: > > We could have a new format, .pya (compiled python archive) that has > > data for many .pyc files in it. [..] > Isn't that essentially what putting the stdlib in a zipfile does? (See

Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?

On 2018-09-18, Carl Shapiro wrote: > How might people feel about using the linker to bundle a list of pre-loaded > modules into a single-file executable? The users of Python are pretty diverse so it depends on who you ask. Some would like a giant executable that includes everything they need (so o

Re: [Python-Dev] dear core-devs

On 2018-10-02, Michael Felt wrote: > I am sorry, for myself obviously - but also for Python. Obviously, I am > doing it all wrong - as I see lots of other issues being picked up > immediately. I'm not sure that's the case. There are a lot of PRs or bugs that sit there without getting reviews. Th

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

On 2018-10-28, Benjamin Peterson wrote: > I don't think more or less API should be magically included based > on whether Py_BUILD_CORE is defined or not. I agree. > If we want to have private headers, we should include them where > needed and not install them. Really, Py_BUILD_CORE should go away

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

On 2018-11-09, Dino Viehland wrote: > Rather than adding yet another pre-processor directive for this I would > suggest just adding a new header file that only has the new stable API. > For example it could just be "py.h" or "pyapi.h". It would have all of the > definitions for the stable API. I

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

On 2018-11-16, Brett Cannon wrote: > I think part of the challenge here (and I believe it has been > brought up elsewhere) is no one knows what kind of API is > necessary for some faster VM other than PyPy. I think we have some pretty good ideas as to what are the problematic parts of the current

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

On 2018-11-16, Nathaniel Smith wrote: > [..] it seems like you should investigate (a) whether you can make > Py_LIMITED_API *be* that API, instead of having two different > ifdefs That might be a good idea. One problem is that we might like to make backwards incompatible changes to Py_LIMITED_API

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

On 2018-11-19, Victor Stinner wrote: > Moreover, I failed to find anyone who can explain me how the C API > is used in the wild, which functions are important or not, what is > the C API, etc. One idea is to download a large sample of extension modules from PyPI and then analyze them with some aut

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

On 2018-11-19, Antoine Pitrou wrote: > There are important use cases for the C API where it is desired to have > fast type-specific access to Python objects such as tuples, ints, > strings, etc. This is relied upon by modules such as _json and _pickle, > and third-party extensions as well. Thank

Re: [Python-Dev] C API changes

On 2018-11-29, Armin Rigo wrote: > ...Also, although I'm discussing it here, I think the whole approach > would be better if done as a third-party extension for now, without > requiring changes to CPython---just use the existing C API to > implement the CPython version. Hello Armin, Thank you for

Re: [Python-Dev] Add more SyntaxWarnings?

On 2019-01-24, Terry Reedy wrote: > Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a > proof-of-concept diff that handled the above and many similar cases. I believe that in general we should give better errors or warnings if we can do it without huge difficulty. Serhiy's patch

Re: [Python-Dev] Asking for reversion

On 2019-02-05, Giampaolo Rodola' wrote: > The main problem I have with this PR is that it seems to introduce > 8 brand new APIs, but since there is no doc, docstrings or tests > it's unclear which ones are supposed to be used, how or whether > they are supposed to supersede or deprecate older (slow

Re: [Python-Dev] Asking for reversion

I wrote: > Could we somehow mark these APIs as experimental in 3.8? It seems the change "e5ef45b8f519a9be9965590e1a0a587ff584c180" the one we are discussing. It adds two new files: Lib/multiprocessing/shared_memory.py Modules/_multiprocessing/posixshmem.c It doesn't introduce new C APIs. S

Re: [Python-Dev] About the future of multi-process Python

On 2019-02-06, Antoine Pitrou wrote: > For maximum synergy between these initiatives and the resulting APIs, > it is better if things are done in the open ;-) Hi Antoine, It would be good if we could have some feedback from alternative Python implementations as well. I suspect they might want to

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

On 2019-02-13, Barry Warsaw wrote: > I personally would like for `python` to be the latest Python 3 > version (or perhaps Brett’s launcher), `python2` to be Python 2.7 > where installed (and not mandatory). `python3` would be an alias > for the latest Python 3. To me, having 'py' on Unix would be

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

On 2019-02-13, Terry Reedy wrote: > It appears python is already python3 for a large majority of human users (as > opposed to machines). IMHO, the question about where /usr/bin/python points is more important for machines than for humans. Think about changing /bin/sh to some different version of

Re: [Python-Dev] Possible performance regression

On 2019-02-25, Eric Snow wrote: > So it looks like commit ef4ac967 is not responsible for a performance > regression. I did a bit of exploration myself and that was my conclusion as well. Perhaps others would be interested in how to use "perf" so I did a little write up: https://discuss.python.o

[Python-Dev] Register-based VM [Was: Possible performance regression]

On 2019-02-26, Victor Stinner wrote: > I made an attempt once and it was faster: > https://faster-cpython.readthedocs.io/registervm.html Interesting. I don't think I have seen that before. Were you aware of "Rattlesnake" before you started on that? It seems your approach is similar. Probably n

[Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

On 2019-02-26, Gregory P. Smith wrote: > On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw wrote: > For an OS distro provided interpreter, being able to restrict its use to > only OS distro provided software would be ideal (so ideal that people who > haven't learned the hard distro maintenance lessons

Re: [Python-Dev] Possible performance regression

On 2019-02-26, Raymond Hettinger wrote: > That said, I'm only observing the effect when building with the > Mac default Clang (Apple LLVM version 10.0.0 (clang-1000.11.45.5). > When building GCC 8.3.0, there is no change in performance. My guess is that the code in _PyEval_EvalFrameDefault() got c

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

On 2019-02-27, Greg Ewing wrote: > Joe Jevnik via Python-Dev wrote: > > If Python switched to a global stack and global registers we may be able > > to eliminate a lot of instructions that just shuffle data from the > > caller's stack to the callee's stack. > > That would make implementing generat

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

On 2019-02-26, Stephane Wirtel wrote: > I also filled an issue [2] for brotlipy (used by httpbin and requests). > The problem is with PyInterpreterState. I tried compiling psycopg2 today and it has a similar problem: psycopg/psycopgmodule.c: In function ‘psyco_is_main_interp’: psycopg/psy

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

On 2019-02-27, Victor Stinner wrote: > The compiler begins with using static single assignment form (SSA) but > then uses a register allocator to reduce the number of used registers. > Usually, at the end you have less than 5 registers for a whole > function. In case anyone is interested on workin

Re: [Python-Dev] PEP 556 threaded garbage collection & linear recursion in gc

On 2019-03-28, Antoine Pitrou wrote: > On Wed, 27 Mar 2019 15:59:25 -0700 > "Gregory P. Smith" wrote: > > > > That had a C++ stack trace 1000+ levels deep repeating the pattern of > > > > ... > > @ 0x564d59bd21de 32 func_dealloc > > @ 0x564d59bce0c1 32 cell_deal

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

On 2019-04-24, Victor Stinner wrote: > The current blocker issue is that the Py_DEBUG define imply the > Py_TRACE_REFS define I think your change to make Py_TRACE_REFS as separate configure flag is fine. I've used the trace fields to debug occasionally but I don't use it often enough to need it e

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

On 2019-04-27, Nathaniel Smith wrote: > For Py_TRACE_REFS specifically, IIUC the only goal is to be able to produce > a list of all live objects on demand. If that's the goal, then static type > objects aren't a huge deal. You can't add extra data into the type objects > themselves, but since there

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

On 2019-05-21, Terry Reedy wrote: > The problem with this argument, taken by itself, it that it would argue for > adding to the stdlib 100s or 1000s of modules or packages that would be > useful to many more people than the modules proposed to be dropped. I don't think it does. We are not talking

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

On 2019-05-21, Brett Cannon wrote: > On Tue., May 21, 2019, 13:07 Neil Schemenauer, > wrote: > > Here is an alternative, straw man, proposal. Split the CPython repo > > into two parts: > > > > - core Python: minimal possible stdlib > > - everythi

[Python-Dev] Re: obmalloc (was Have a big machine and spare time? Here's a possible Python bug.)

On 2019-06-06, Tim Peters wrote: > Like now: if the size were passed in, obmalloc could test the size > instead of doing the `address_in_range()` dance(*). But if it's ever > possible that the size won't be passed in, all the machinery > supporting `address_in_range()` still needs to be there, an

[Python-Dev] Re: obmalloc (was Have a big machine and spare time? Here's a possible Python bug.)

On 2019-06-06, Tim Peters wrote: > The doubly linked lists in gc primarily support efficient > _partitioning_ of objects for gc's purposes (a union of disjoint sets, > with constant-time moving of an object from one set to another, and > constant-time union of disjoint sets). "All objects" is almo

[Python-Dev] Re: Expected stability of PyCode_New() and types.CodeType() signatures

On 2019-05-31, Simon Cross wrote: > As the maintainer of Genshi, one the libraries affected by the CodeType and > similar changes, I thought I could add a users perspective to the > discussion: [...] Thanks. I think this change to PyCode_New() could have been handled a bit better. Couldn't we in

[Python-Dev] Re: obmalloc (was Have a big machine and spare time? Here's a possible Python bug.)

On 2019-06-09, Tim Peters wrote: > And now there's a PR that removes obmalloc's limit on pool sizes, and, > for a start, quadruples pool (and arena!) sizes on 64-bit boxes: Neat. > As the PR says, > > """ > It would be great to get feedback from 64-bit apps that do massive > amounts of small-obj

[Python-Dev] radix tree arena map for obmalloc

I've been working on this idea for a couple of days. Tim Peters has being helping me out and I think it has come far enough to get some more feedback. It is not yet a good replacement for the current address_in_range() test. However, performance wise, it is very close. Tim figures we are not do

[Python-Dev] Re: radix tree arena map for obmalloc

On 2019-06-15, Inada Naoki wrote: > Oh, do you mean your branch doesn't have headers in each page? That's right. Each pool still has a header but pools can be larger than the page size. Tim's obmalloc-big-pool idea writes something to the head of each page within a pool. The radix tree doesn't

[Python-Dev] Re: radix tree arena map for obmalloc

On 2019-06-14, Tim Peters wrote: > However, last I looked there Neil was still using 4 KiB obmalloc > pools, all page-aligned. But using much larger arenas (16 MiB, 16 > times bigger than my branch, and 64 times bigger than Python currently > uses). I was testing it verses your obmalloc-big-pool

[Python-Dev] Re: radix tree arena map for obmalloc

Here are benchmark results for 64 MB arenas and 16 kB pools. I ran without the --fast option and on a Linux machine in single user mode. The "base" columm is the obmalloc-big-pools branch with ARENA_SIZE = 64 MB and POOL_SIZE = 16 kB. The "radix" column is obmalloc_radix_tree (commit 5e00f6041)

[Python-Dev] Re: radix tree arena map for obmalloc

On 2019-06-15, Antoine Pitrou wrote: > We should evaluate what problem we are trying to solve here, instead > of staring at micro-benchmark numbers on an idle system. I think a change to obmalloc is not going to get accepted unless we can show it doesn't hurt these micro-benchmarks. To displace t

[Python-Dev] Re: radix tree arena map for obmalloc

On 2019-06-15, Tim Peters wrote: > At the start, obmalloc never returned arenas to the system. The vast > majority of users were fine with that. Yeah, I was totally fine with that back in the day. However, I wonder now if there is a stronger reason to try to free memory back to the OS. Years ag

[Python-Dev] Re: obmalloc (was Have a big machine and spare time? Here's a possible Python bug.)

On 2019-06-21, Tim Peters wrote: > [Thomas Wouters ] > > Getting rid of address_in_range sounds like a nice idea, and I > > would love to test how feasible it is -- I can run such a change > > against a wide selection of code at work, including a lot of > > third-party extension modules, but I don'

[Python-Dev] Re: obmalloc (was Have a big machine and spare time? Here's a possible Python bug.)

For those who would like to test with something compatible with Python 3.7.3, I made re-based branches here: https://github.com/nascheme/cpython/tree/obmalloc_radix_v37 https://github.com/nascheme/cpython/tree/obmalloc_big_pools_v37 They should be ABI compatible with Python 3.7.3. So,

[Python-Dev] Re: Removing dead bytecode vs reporting syntax errors

On 2019-07-06, Victor Stinner wrote: > More people seems to expect "if 0: ..." to be removed, than people who > care of syntax errors on "if 0". One small data point: I have shipped code that depended on 'if 0' removing code from the .pyc file. The code inside was not meant to be released publicl

[Python-Dev] Re: Optimizing pymalloc (was obmalloc

On 2019-07-09, Inada Naoki wrote: > So I tried to use LIKELY/UNLIKELY macro to teach compiler hot part. > But I need to use > "static inline" for pymalloc_alloc and pymalloc_free yet [1]. I think LIKELY/UNLIKELY is not helpful if you compile with LTO/PGO enabled. So, I would try that first. Also

[Python-Dev] Re: Optimizing pymalloc (was obmalloc

On 2019-07-09, Inada Naoki wrote: > PyObject_Malloc inlines pymalloc_alloc, and PyObject_Free inlines > pymalloc_free. > But compiler doesn't know which is the hot part in pymalloc_alloc and > pymalloc_free. Hello Inada, I don't see this on my PC. I'm using GCC 8.3.0. I have configured the bui

[Python-Dev] Re: What to do about invalid escape sequences

Making it an error so soon would be mistake, IMHO. That will break currently working code for small benefit. When Python was a young language with a few thousand users, it was easier to make these kinds of changes. Now, we should be much more conservative and give people a long time and a lot

[Python-Dev] Re: How to extricate large set of diffs from hg.python.org/sandbox?

On 2019-08-07, Skip Montanaro wrote: > Victor's experiments into a register-based virtual machine live here: > > https://hg.python.org/sandbox/registervm > > I'd like to revive them, if for no other reason to understand what he > did. I see no obvious way to collect them all as a massive diff. I

[Python-Dev] Re: Pattern matching reborn: PEP 622 is dead, long live PEP 634, 635, 636

Woah, this is both exciting and scary.  If adopted, it will be a major change to how Python programs are written.  It seems a lot of work has been put into polishing the design.  That's good because if we do this, will not be easy to fix things later if we made design errors. One of my first t

[Python-Dev] Re: PEP: Deferred Evaluation Of Annotations Using Descriptors

On 2021-01-11, Łukasz Langa wrote: > The stringification process which your PEP describes as costly > only happens during compilation of a .py file to .pyc. Since > pip-installing pre-compiles modules for the user at installation > time, there is very little runtime penalty for a fully annotated >

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

On 2021-01-12, Victor Stinner wrote: > It seems like a safer approach is to continue the work on > bpo-40077: "Convert static types to PyType_FromSpec()". I agree that trying to convert static types is a good idea. Another possible bonus might be that we can gain some performance by integrating g

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

On 2021-01-12, Pablo Galindo Salgado wrote: > One worry that I have in general with this move is the usage of > _PyType_GetModuleByDef to get the type object from the module > definition. This normally involves getting a TLS in every instance > creation, which can impact notably performance for som

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

On 2021-01-12, Petr Viktorin wrote: > Unfortunately, it's not just the creation that needs to be changed. > You also need to decref Foo_Type somewhere. Add the type to the module dict? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe s

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

On 2021-05-07, Pablo Galindo Salgado wrote: > Technically the main concern may be the size of the unmarshalled > pyc files in memory, more than the storage size of disk. It would be cool if we could mmap the pyc files and have the VM run code without an unmarshal step. One idea is something simil

[Python-Dev] Difficulty of testing beta releases now available

On 2021-05-04, Łukasz Langa wrote: > We strongly encourage maintainers of third-party Python projects > to test with 3.10 during the beta phase and report issues found to > the Python bug tracker as soon as > possible. Testing with Python 3.10b1 is not easy, at least for

[Python-Dev] Smoothing the transition from Python 2 to 3

[I've posted something about this on python-ideas but since I now have some basic working code, I think it is more than an idea.] I think the uptake of Python 3 is starting to accelerate. That's good. However, there are still millions or maybe billions of lines of Python code that still needs to

Re: [Python-Dev] Smoothing the transition from Python 2 to 3

On 2016-06-09, Brett Cannon wrote: > On Thu, 9 Jun 2016 at 14:56 Nick Coghlan wrote: > > Once you switch to those now recommended more conservative migration > > tools, the tool suite you request already exists: > > > > - update your code with modernize or futurize > > - check it still runs on Pyt

Re: [Python-Dev] Smoothing the transition from Python 2 to 3

On 2016-06-09, Brett Cannon wrote: > I don't think you meant for what you said to sound insulting, > Neil, but it did feel like it upon first reading. Sorry, I think I misunderstood what you and Nick were saying. I've experienced a fair amount of negative feedback on my idea so I'm pretty cranky

Re: [Python-Dev] Smoothing the transition from Python 2 to 3

On 6/10/2016 10:49 AM, Nick Coghlan wrote: What Brett said is mostly accurate for me, except with one slight caveat: I've been explicitly trying to nudge you towards making the *existing tools better*, rather than introducing new tools. With modernize and futurize we have a fairly clear trade-off

Re: [Python-Dev] Smoothing the transition from Python 2 to 3

Nick Coghlan wrote: > It could be very interesting to add an "ascii-warn" codec to Python > 2.7, and then set that as the default encoding when the -3 flag is > set. I don't think that can work. The library code in Python would spew out warnings even in the cases when nothing is wrong with the a

  1   2   3   >