Re: [Python-Dev] Aligning the packaging.python.org theme with the rest of the docs

On 30 May 2017 at 22:08, Antoine Pitrou wrote: > On Tue, 30 May 2017 21:49:19 +1000 > Nick Coghlan wrote: >> >> Here's an alternate wording for the README that would focus on those >> considerations without explicitly asking folks not to use the theme: >>

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

On 06/01/2017 02:50 AM, Antoine Pitrou wrote: Another possible strategy is: allocate several arenas at once (using a larger mmap() call), and use MADV_DONTNEED to relinquish individual arenas. Thus adding a *fourth* layer of abstraction over memory we get from the OS? block -> pool ->

[Python-Dev] 2017 Python Language Summit coverage -- Round 2

Hola python-dev, Thanks for all the positive feedback on the coverage (and the corrections/clarifications in the comments too)! There is, it seems, always more to do, but I do have three additional articles from the summit up now and should complete the coverage over the next week. The

Re: [Python-Dev] PEP 7 and braces { .... } on if

https://github.com/python/peps/pull/280/files On Jun 01, 2017, at 09:08 PM, Brett Cannon wrote: >If you create an issue at github.com/python/peps and assign it to me I will >get to it someday. :) pgpqhM6HQldC5.pgp Description: OpenPGP digital signature

Re: [Python-Dev] PEP 7 and braces { .... } on if

If you create an issue at github.com/python/peps and assign it to me I will get to it someday. :) On Thu, 1 Jun 2017 at 00:19 Victor Stinner wrote: > 2017-05-31 19:27 GMT+02:00 Guido van Rossum : > > I interpret the PEP (...) > > Right, the phrasing

Re: [Python-Dev] "Global freepool"

2017-06-01 22:16 GMT+02:00 Serhiy Storchaka : > The issue [1] still is open. Patches neither applied nor rejected. They > exposes the speed up in microbenchmarks, but it is not large. Up to 40% for > iterating over enumerate() and 5-7% for hard integer computations like >

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

On Thursday 01 June 2017 01:27 PM, Victor Stinner wrote: > The GNU libc malloc uses a variable threshold to choose between sbrk() > (heap memory) or mmap(). It starts at 128 kB or 256 kB, and then is > adapted depending on the workload (I don't know how exactly). The threshold starts at 128K and

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On 01Jun2017 1010, Nathaniel Smith wrote: I believe that for answering this question about the ssl module, it's really only Linux users that matter, since pip/requests/everyone else pushing for this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS (IIUC) is to stop using

Re: [Python-Dev] "Global freepool"

01.06.17 21:44, Larry Hastings пише: p.s. Speaking of freelists, at one point Serhiy had a patch adding a freelist for single- and I think two-digit ints. Right now the only int creation optimization we have is the array of constant "small ints"; if the int you're constructing isn't one of

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On Jun 1, 2017, at 3:57 PM, Donald Stufft wrote: > > Note that on macOS, within the next year macOS users using the system Python > are going to be unable to talk to PyPI anyways (unless Apple does something > here, which I think they will), but in either case, Apple was

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On Jun 1, 2017, at 3:51 PM, Paul Moore wrote: > > Linux users often use the OS-supplied Python, and so getting the > distributions to upgrade, and to backport upgrades to old versions of > their OS and (push those backports as required updates) is the route > to get the

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On Jun 1, 2017, at 1:09 PM, Barry Warsaw wrote: > > On Jun 02, 2017, at 02:14 AM, Chris Angelico wrote: > >> But it also includes people on stable Linux distros, where they have >> automatic updates provided by Red Hat or Debian or whomever, so a change like >> this WILL

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Thu, Jun 01, 2017 at 04:01:54PM +0100, Cory Benfield wrote: > > lower net cost than getting the world into a situation where pip no > > longer runs on the >1e6 EC2 instances that will be running Ubuntu > > 14.04/16.04 LTS until the turn of the next decade. > So for the record I’m assuming

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On 1 June 2017 at 17:14, Chris Angelico wrote: > Python 2 users include people on Windows who install it themselves, > and then have no mechanism for automatic updates. They'll probably > stay on whatever 2.7.x they first got, until something forces them to > update. But it also

Re: [Python-Dev] "Global freepool"

On 06/01/2017 02:20 AM, Victor Stinner wrote: I would like to understand how private free lists are "so much" faster. In fact, I don't recall if someone *measured* the performance speedup of these free lists :-) I have, recently, kind of by accident. When working on the Gilectomy I turned

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Jun 01, 2017, at 07:22 PM, Victor Stinner wrote: >I can help Canonical to backport MemoryBIO *if they want* to >cherry-pick this feature ;-) (Pedantically speaking, this falls under the Ubuntu project's responsibility, not directly Canonical.) Writing the patch is only part of the process:

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

On 06/01/2017 02:03 AM, Victor Stinner wrote: 2017-06-01 10:41 GMT+02:00 Larry Hastings : On 06/01/2017 01:19 AM, Antoine Pitrou wrote: If you'd like to go that way anyway, I would suggest 1MB as a starting point in 3.7. I understand the desire for caution. But I was

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-01 19:10 GMT+02:00 Nathaniel Smith : > (...) since pip/requests/everyone else pushing for > this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS > (IIUC) is to stop using the ssl module entirely in favor of new ctypes > bindings for their respective

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-01 19:09 GMT+02:00 Barry Warsaw : > By 2020, only Ubuntu 16.04 and 18.04 will still be maintained, so while 18.04 > will likely contain whatever the latest 2.7 is available at that time, 16.04 > won't track upstream point releases, but instead will get select cherry >

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Jun 1, 2017 9:20 AM, "Chris Angelico" wrote: On Fri, Jun 2, 2017 at 1:01 AM, Cory Benfield wrote: > The answer to that is honestly not clear to me. I chatted with the pip developers, and they have 90%+ of their users currently on Python 2, but more than

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Jun 02, 2017, at 02:14 AM, Chris Angelico wrote: >But it also includes people on stable Linux distros, where they have >automatic updates provided by Red Hat or Debian or whomever, so a change like >this WILL propagate - particularly (a) as the window is three entire years, >and (b) if the

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-01 18:51 GMT+02:00 Chris Angelico : > Hmm. So it's really hard to know. Pity. I suppose it's too much to ask > for IP-based stat exclusion for the most commonly-used CI systems > (Travis, Circle, etc)? Still, it does look like most pip usage happens > on Linux. Also, it

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Fri, Jun 2, 2017 at 2:35 AM, Cory Benfield wrote: > I have figures for the download numbers, which are an awkward proxy because > most people don’t CI on Windows and macOS, but they’re the best we have. > Linux has approximately 20x the download numbers of either Windows

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On 1 Jun 2017, at 17:14, Chris Angelico wrote: > > > Do you have figures for how many people use pip on Windows vs Linux vs Mac OS? I have figures for the download numbers, which are an awkward proxy because most people don’t CI on Windows and macOS, but they’re the best

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Fri, Jun 2, 2017 at 1:01 AM, Cory Benfield wrote: > The answer to that is honestly not clear to me. I chatted with the pip > developers, and they have 90%+ of their users currently on Python 2, but more > than half of those are on 2.7.9 or later. This shows some interest

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Thu, 1 Jun 2017 15:09:41 +0100 Cory Benfield wrote: > > As to the plans to move Requests to a more event loop-y model, I think that > it does stand in the way of this, but only insomuch as, again, we want our > event loopy model to be as bug-free as possible. But I can

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On 1 Jun 2017, at 15:10, David Wilson wrote: > Finding someone to audit the signature checking capabilities of [0] will > have vastly lower net cost than getting the world into a situation where > pip no longer runs on the >1e6 EC2 instances that will be running Ubuntu

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

It seems very complex and not portable at all to "free" a part of an arena. We already support freeing a whole arena using munmap(). I was a huge enhancement in our memory allocator. Change made in Python 2.5? I don't recall, ask Evan Jones: http://www.evanjones.ca/memoryallocator/ :-) I'm not

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

On Thursday 01 June 2017 01:53 PM, INADA Naoki wrote: > * On Linux, madvice(..., MADV_DONTNEED) can be used. madvise does not reduce the commit charge in the Linux kernel, so in high consumption scenarios (and where memory overcommit is disabled or throttled) you'll see programs dying with OOM

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

Trying to transfer github comments from https://github.com/python/peps/pull/272#pullrequestreview-41388700: I said: > Tornado has been doing TLS in an event-loop model in python 2.5+ with just wrap_socket, no MemoryBIO necessary. What am I missing? MemoryBIO certainly gives some extra

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Thu, Jun 01, 2017 at 12:18:48PM +0100, Cory Benfield wrote: > So, this will work, but at a performance and code cleanliness cost. > This essentially becomes a Python-2-only code-path, and a very large > and complex one at that. "Doctor, it hurts when I do this .." Fine, then how about rather

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On 1 Jun 2017, at 14:53, Antoine Pitrou wrote: > > On Thu, 1 Jun 2017 14:37:55 +0100 > Cory Benfield wrote: >>> >>> And indeed it doesn't. Unless the target user base for pip is widely >>> different than Python's, it shouldn't cause you any problems

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Thu, 1 Jun 2017 14:37:55 +0100 Cory Benfield wrote: > > > > And indeed it doesn't. Unless the target user base for pip is widely > > different than Python's, it shouldn't cause you any problems either. > > Maybe not now, but I think it’s fair to say that it did, right?

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On 1 Jun 2017, at 14:21, Antoine Pitrou wrote: > > > Le 01/06/2017 à 15:12, Cory Benfield a écrit : >> >> I don’t know what to do with that answer, really. I gave you some data (80%+ >> of requests downloads over the last month were Python 2), and you responded >> with

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

Le 01/06/2017 à 15:12, Cory Benfield a écrit : > > I don’t know what to do with that answer, really. I gave you some data (80%+ > of requests downloads over the last month were Python 2), and you responded > with “it doesn’t cause us problems”. And indeed it doesn't. Unless the target user

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On 1 Jun 2017, at 12:20, Antoine Pitrou wrote: > > On Thu, 1 Jun 2017 12:01:41 +0100 > Cory Benfield wrote: >> In principle, sure. In practice, that means most of our users don’t use >> those features and so we don’t get any feedback on whether

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Thu, 1 Jun 2017 12:01:41 +0100 Cory Benfield wrote: > In principle, sure. In practice, that means most of our users don’t use those > features and so we don’t get any feedback on whether they’re good solutions > to the problem. On bugs.python.org we get plenty of feedback

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On 1 Jun 2017, at 12:09, David Wilson wrote: > > On Thu, Jun 01, 2017 at 11:47:31AM +0100, Cory Benfield wrote: > >> I have, but discarded the idea. > > I'm glad to hear it was given sufficent thought. :) > > I have one final 'crazy' idea, and actually it does not

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Thu, Jun 01, 2017 at 11:47:31AM +0100, Cory Benfield wrote: > I have, but discarded the idea. I'm glad to hear it was given sufficent thought. :) I have one final 'crazy' idea, and actually it does not seem to bad at all: can't you just fork a subprocess or spawn threads to handle the

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On 1 Jun 2017, at 11:51, Antoine Pitrou wrote: > > On Thu, 1 Jun 2017 11:45:14 +0100 > Cory Benfield wrote: >> >> I am claiming that using OpenSSL certificate validation with root stores >> that are not intended for OpenSSL can be. This is because

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On 1 Jun 2017, at 11:39, Antoine Pitrou wrote: > > On Thu, 1 Jun 2017 11:22:21 +0100 > Cory Benfield wrote: >> >> Who is the “we” that should move on? Python core dev? Or the Python >> ecosystem? > > Sorry. Python core dev certainly. As for the

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Thu, 1 Jun 2017 11:45:14 +0100 Cory Benfield wrote: > > I am claiming that using OpenSSL certificate validation with root stores that > are not intended for OpenSSL can be. This is because trust of a certificate > is non-binary. For example, consider WoSign. The Windows

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On 1 Jun 2017, at 11:39, David Wilson wrote: > > Hi Cory, > > On Thu, Jun 01, 2017 at 11:22:21AM +0100, Cory Benfield wrote: > >> We want to support as many TLS backends as possible. > > Just a wild idea, but have you investigated a pure-Python fallback for > 2.7

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On 1 Jun 2017, at 11:28, Antoine Pitrou wrote: > > > Le 01/06/2017 à 12:23, Cory Benfield a écrit : >> >> No it can’t. >> >> OpenSSL builds chains differently, and disregards some metadata that Windows >> and macOS store, which means that cert validation will work

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Thu, 1 Jun 2017 11:22:21 +0100 Cory Benfield wrote: > > Who is the “we” that should move on? Python core dev? Or the Python ecosystem? Sorry. Python core dev certainly. As for the rest of the ecosystem, it is moving on as well. > Requests is stuck in a place from which

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

Hi Cory, On Thu, Jun 01, 2017 at 11:22:21AM +0100, Cory Benfield wrote: > We want to support as many TLS backends as possible. Just a wild idea, but have you investigated a pure-Python fallback for 2.7 such as TLSlite? Of course the fallback need only be used during bootstrapping, and the

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On 1 Jun 2017, at 11:18, Antoine Pitrou wrote: > > On Thu, 1 Jun 2017 20:05:48 +1000 > Chris Angelico wrote: >> >> As stated in this thread, OS-provided certificates are not handled by >> that. For instance, if a local administrator distributes a

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

Le 01/06/2017 à 12:23, Cory Benfield a écrit : > > No it can’t. > > OpenSSL builds chains differently, and disregards some metadata that Windows > and macOS store, which means that cert validation will work differently than > in the system store. This can lead to pip accepting a cert marked

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

> On 1 Jun 2017, at 10:23, Antoine Pitrou wrote: > > Yes, I disagree. We needn't backport that new API to Python 2.7. > Perhaps it's time to be reasonable: Python 2.7 has been in bugfix-only > mode for a very long time. Python 3.6 is out. We should move on. Who is the

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Thu, 1 Jun 2017 20:05:48 +1000 Chris Angelico wrote: > > As stated in this thread, OS-provided certificates are not handled by > that. For instance, if a local administrator distributes a self-signed > cert for the intranet server, web browsers will use it, but pip will >

Re: [Python-Dev] "Global freepool"

01.06.17 12:20, Victor Stinner пише: 2017-06-01 10:40 GMT+02:00 Antoine Pitrou : This is already exactly how PyObject_Malloc() works. (...) Oh ok, good to know... IMHO the main thing the private freelists have is that they're *private* precisely, so they can avoid a

Re: [Python-Dev] "Global freepool"

I thought pymalloc is SLAB allocator. What is the difference between SLAB and pymalloc allocator? On Thu, Jun 1, 2017 at 6:20 PM, Victor Stinner wrote: > 2017-06-01 10:40 GMT+02:00 Antoine Pitrou : >> This is already exactly how PyObject_Malloc()

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Thu, Jun 1, 2017 at 8:01 PM, Antoine Pitrou wrote: > On Thu, 1 Jun 2017 19:50:22 +1000 > Chris Angelico wrote: >> On Thu, Jun 1, 2017 at 7:23 PM, Antoine Pitrou wrote: >> >> Do you also disagree on the need of the need of the PEP 546

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Thu, 1 Jun 2017 19:50:22 +1000 Chris Angelico wrote: > On Thu, Jun 1, 2017 at 7:23 PM, Antoine Pitrou wrote: > >> Do you also disagree on the need of the need of the PEP 546 > >> (backport) to make the PEP 543 (new TLS API) feasible in practice? > > > >

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Thu, Jun 1, 2017 at 7:23 PM, Antoine Pitrou wrote: >> Do you also disagree on the need of the need of the PEP 546 >> (backport) to make the PEP 543 (new TLS API) feasible in practice? > > Yes, I disagree. We needn't backport that new API to Python 2.7. > Perhaps it's time

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

On Thu, 1 Jun 2017 18:37:17 +0900 INADA Naoki wrote: > x86's hugepage is 2MB. > And some Linux enables "Transparent Huge Page" feature. > > Maybe, 2MB arena size is better for TLB efficiency. > Especially, for servers having massive memory. But, since Linux is able to

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

For the ARENA_SIZE, will that be better to setting by ./configure first, and without hard code in c files? 2017-06-01 17:37 GMT+08:00 INADA Naoki : > x86's hugepage is 2MB. > And some Linux enables "Transparent Huge Page" feature. > > Maybe, 2MB arena size is better for

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

x86's hugepage is 2MB. And some Linux enables "Transparent Huge Page" feature. Maybe, 2MB arena size is better for TLB efficiency. Especially, for servers having massive memory. On Thu, Jun 1, 2017 at 4:38 PM, Larry Hastings wrote: > > > When CPython's small block allocator

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

On Thu, Jun 1, 2017 at 10:45 AM, Larry Hastings wrote: > On 06/01/2017 01:41 AM, Larry Hastings wrote: > > On 06/01/2017 01:19 AM, Antoine Pitrou wrote: > > malloc() you said? Arenas are allocated using mmap() nowadays, right? > > malloc() and free(). See

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

Le 01/06/2017 à 11:13, Victor Stinner a écrit : > That's why pip uses Requests which uses certifi (Mozilla > bundled root certificate authorities.) pip could use certifi without using Requests. My guess is that Requests is used mostly because it eases coding. > pip and so Requests are part of

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

Thanks for detailed info. But I don't think it's a big problem. Arenas are returned to system by chance. So other processes shouldn't relying to it. And I don't propose to stop returning arena to system. I just mean per pool (part of arena) MADV_DONTNEED can reduce RSS. If we use very large

Re: [Python-Dev] "Global freepool"

2017-06-01 10:40 GMT+02:00 Antoine Pitrou : > This is already exactly how PyObject_Malloc() works. (...) Oh ok, good to know... > IMHO the main thing the > private freelists have is that they're *private* precisely, so they can > avoid a couple of conditional branches. I

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

> * On Linux, madvice(..., MADV_DONTNEED) can be used. Recent Linux has MADV_FREE. It is faster than MADV_DONTNEED, https://lwn.net/Articles/591214/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-01 10:57 GMT+02:00 Antoine Pitrou : > If Requests is to remain 2.7-compatible, it's up to Requests to do the > necessary work to do so. In practice, CPython does include Requests in ensurepip. Because of that, it means that Requests cannot use any C extension. CPython

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

2017-06-01 10:41 GMT+02:00 Larry Hastings : > On 06/01/2017 01:19 AM, Antoine Pitrou wrote: > If you'd like to go that way anyway, I would suggest 1MB as a starting > point in 3.7. > > I understand the desire for caution. But I was hoping maybe we could > experiment with 4mb

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

On Wed, 31 May 2017 14:09:20 -0600 Jim Baker wrote: > > But I object to a completely new feature being added to 2.7 to support the > implementation of event loop SSL usage. This feature cannot be construed as > a security fix, and therefore does not qualify as a feature

Re: [Python-Dev] "Global freepool"

Hi, As you said, I think PyObject_Malloc() is fast enough. But PyObject_Free() is somewhat complex. Actually, there are some freelists (e.g. tuple, dict, frame) and they improve performance significantly. My "global unified freelist" idea is unify them. The merit is: * Unify

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

On Thu, 1 Jun 2017 01:41:15 -0700 Larry Hastings wrote: > On 06/01/2017 01:19 AM, Antoine Pitrou wrote: > > If you'd like to go that way anyway, I would suggest 1MB as a starting > > point in 3.7. > > I understand the desire for caution. But I was hoping maybe we could >

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

On 06/01/2017 01:41 AM, Larry Hastings wrote: On 06/01/2017 01:19 AM, Antoine Pitrou wrote: malloc() you said? Arenas are allocated using mmap() nowadays, right? malloc() and free(). See _PyObject_ArenaMalloc (etc) in Objects/obmalloc.c. Oh, sorry, I forgot how to read. If ARENAS_USE_MMAP

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

On 06/01/2017 01:19 AM, Antoine Pitrou wrote: If you'd like to go that way anyway, I would suggest 1MB as a starting point in 3.7. I understand the desire for caution. But I was hoping maybe we could experiment with 4mb in trunk for a while? We could change it to 1mb--or even 256k--before

[Python-Dev] "Global freepool"

On Thu, 1 Jun 2017 09:57:04 +0200 Victor Stinner wrote: > > By the way, Naoki INADA also proposed a different idea: > > "Global freepool: Many types has it’s own freepool. Sharing freepool > can increase memory and cache efficiency. Add PyMem_FastFree(void* > ptr,

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

2017-06-01 10:23 GMT+02:00 INADA Naoki : > AFAIK, allocating arena doesn't eat real (physical) memory. > > * On Windows, VirtualAlloc is used for arena. Real memory page is assigned > when the page is used first time. > * On Linux and some other *nix, anonymous mmap is

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

2017-06-01 10:19 GMT+02:00 Antoine Pitrou : > Yes, this is the same kind of reason the default page size is still 4KB > on many platforms today, despite typical memory size having grown by a > huge amount. Apart from the cost of fragmentation as you mentioned, > another issue

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

Hello. AFAIK, allocating arena doesn't eat real (physical) memory. * On Windows, VirtualAlloc is used for arena. Real memory page is assigned when the page is used first time. * On Linux and some other *nix, anonymous mmap is used. Real page is assigned when first touch, like Windows.

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

On Thu, 1 Jun 2017 00:38:09 -0700 Larry Hastings wrote: > * CPython programs would use more memory. How much? Hard to say. It > depends on their allocation strategy. I suspect most of the time it > would be < 3mb additional memory. But for pathological

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

On 06/01/2017 12:57 AM, Victor Stinner wrote: I would prefer to have an adaptative arena size. For example start at 256 kB and then double the arena size while the memory usage grows. The problem is that pymalloc is currently designed for a fixed arena size. I have no idea how hard it would be

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

2017-06-01 9:38 GMT+02:00 Larry Hastings : > When CPython's small block allocator was first merged in late February 2001, > it allocated memory in gigantic chunks it called "arenas". These arenas > were a massive 256 KILOBYTES apiece. The arena size defines the strict minimum

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

On 06/01/2017 12:38 AM, Larry Hastings wrote: I propose we make the arena size larger. By how much? I asked Victor to run some benchmarks with arenas of 1mb, 2mb, and 4mb. The results with 1mb and 2mb were mixed, but his benchmarks with a 4mb arena size showed measurable (>5%) speedups on

[Python-Dev] The untuned tunable parameter ARENA_SIZE

When CPython's small block allocator was first merged in late February 2001, it allocated memory in gigantic chunks it called "arenas". These arenas were a massive 256 KILOBYTES apiece. This tunable parameter has not been touched in the intervening 16 years. Yet CPython's memory

Re: [Python-Dev] PEP 7 and braces { .... } on if

2017-05-31 19:27 GMT+02:00 Guido van Rossum : > I interpret the PEP (...) Right, the phrasing requires to "interpret" it :-) > (...) as saying that you should use braces everywhere but not > to add them in code that you're not modifying otherwise. (I.e. don't go on a >

Re: [Python-Dev] PEP 7 and braces { .... } on if

01.06.17 09:36, Benjamin Peterson пише: Modern GCC can defend against these kinds of problems. If I introduce a "goto fail" bug somewhere in Python, I get a nice warning: ../Objects/abstract.c: In function ‘PyObject_Type’: ../Objects/abstract.c:35:5: warning: this ‘if’ clause does not guard...

Re: [Python-Dev] PEP 7 and braces { .... } on if

31.05.17 20:27, Guido van Rossum пише: I interpret the PEP as saying that you should use braces everywhere but not to add them in code that you're not modifying otherwise. (I.e. don't go on a brace-adding rampage.) If author and reviewer of a PR disagree I would go with "add braces" since

Re: [Python-Dev] PEP 7 and braces { .... } on if

Modern GCC can defend against these kinds of problems. If I introduce a "goto fail" bug somewhere in Python, I get a nice warning: ../Objects/abstract.c: In function ‘PyObject_Type’: ../Objects/abstract.c:35:5: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation] if (o ==