Re: [Python-Dev] PEP 509: Add a private version to dict (version 3)

Hi, Guido van Rossum and Jim J. Jewett suggested me to *not require* to always increase the dict version if a dict method does not modify its content. I modified the Changes section to only require that the version is increased when the dictionary content is modified. I also explained the nice

Re: [Python-Dev] PEP 509: Add a private version to dict (version 3)

Hi, > Backwards Compatibility > === > > Since the ``PyDictObject`` structure is not part of the stable ABI and > the new dictionary version not exposed at the Python scope, changes are > backward compatible. My current implementation inserts the new ma_version_tag field in

[Python-Dev] PEP 509: Add a private version to dict (version 3)

n, Google Summer of Code 2005 project by Nick Smallbone. This project has a patch for CPython 2.4 which adds ``key_time`` and ``value_time`` fields to dictionary entries. It uses a global process-wide counter for dictionaries, incremented each time that a dictionary is modified. The tim

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-21 2:54 GMT+01:00 Andrew Barnert : > This idea worries me. I'm not sure why, but I think because of threading. > After all, it's pretty rare for two threads to both want to work on the same > dict, but very, very common for two threads to both want to work on _any_ >

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-21 6:08 GMT+01:00 Yury Selivanov : > Yeah, I think that's what we agreed on: > https://mail.python.org/pipermail/python-dev/2016-January/142837.html > > The only advantage of ma_extra pointer is that it allows to add more stuff > to dicts in the future. I don't

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-21 3:34 AM, Victor Stinner wrote: [..] But if a global counter doesn't make the slow more complex and opens new kinds of optimization, I agree to change my PEP 509. Please. That would allow us to use ma_version to implement caches in CPython itself. Yury

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 8:54 PM, Andrew Barnert via Python-Dev wrote: >I think Glenn was assuming we had a single, global version # that all dicts shared without having a per-dict version ID. The key thing here is that we have a global counter that tracks the number of mutations for all dictionaries

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wednesday, January 20, 2016 4:10 PM, Brett Cannon wrote: >I think Glenn was assuming we had a single, global version # that all dicts >shared without having a per-dict version ID. The key thing here is that we >have a global counter that tracks the number of mutations for

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 5:01 PM, Greg Ewing wrote: Yury Selivanov wrote: What I propose is to add a pointer "ma_extra" (same 64bits), which will be set to NULL for most dict instances (instead of ma_version). "ma_extra" can then point to a struct that has a globally unique dict ID (uint64), and a

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, 20 Jan 2016, 17:54 Andrew Barnert wrote: > On Wednesday, January 20, 2016 4:10 PM, Brett Cannon > wrote: > > > >I think Glenn was assuming we had a single, global version # that all > dicts shared without having a per-dict version ID. The key thing

Re: [Python-Dev] PEP 509: Add a private version to dict

Andrew Barnert via Python-Dev wrote: imagine someone manages to remove the GIL from CPython by using STM: now most transactions are bumping that global counter, meaning most transactions fail and have to be retried, If this becomes a problem, the tag could be split into two parts of m and n

Re: [Python-Dev] PEP 509: Add a private version to dict

The easiest version is to have global numbering (as opposed to local). Anyway, I would strongly suggest getting some benchmarks done and showing performance benefits first, because you don't want PEPs to be final when you don't exactly know the details. On Wed, Jan 20, 2016 at 7:02 PM, Yury

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, Jan 20, 2016 at 7:22 PM, Brett Cannon wrote: > > > On Wed, 20 Jan 2016 at 10:11 Yury Selivanov wrote: >> >> On 2016-01-18 5:43 PM, Victor Stinner wrote: >> > Is someone opposed to this PEP 509? >> > >> > The main complain was the change on the

Re: [Python-Dev] PEP 509: Add a private version to dict

Brett, On 2016-01-20 1:22 PM, Brett Cannon wrote: On Wed, 20 Jan 2016 at 10:11 Yury Selivanov > wrote: On 2016-01-18 5:43 PM, Victor Stinner wrote: > Is someone opposed to this PEP 509? > > The main complain was the

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-18 5:43 PM, Victor Stinner wrote: Is someone opposed to this PEP 509? The main complain was the change on the public Python API, but the PEP doesn't change the Python API anymore. I'm not aware of any remaining issue on this PEP. Victor, I've been experimenting with the PEP to

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, 20 Jan 2016 at 10:11 Yury Selivanov wrote: > On 2016-01-18 5:43 PM, Victor Stinner wrote: > > Is someone opposed to this PEP 509? > > > > The main complain was the change on the public Python API, but the PEP > > doesn't change the Python API anymore. > > > > I'm

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, Jan 20, 2016 at 8:00 PM, Yury Selivanov wrote: > > > On 2016-01-20 1:36 PM, Maciej Fijalkowski wrote: >> >> On Wed, Jan 20, 2016 at 7:22 PM, Brett Cannon wrote: >>> >>> >>> On Wed, 20 Jan 2016 at 10:11 Yury Selivanov

Re: [Python-Dev] PEP 509: Add a private version to dict

On 1/20/2016 10:36 AM, Maciej Fijalkowski wrote: Why can't you simply use the id of the dict object as the globally unique >dict ID? It's already globally unique amongst all Python objects which makes >it inherently unique amongst dicts. > >___

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 1:36 PM, Maciej Fijalkowski wrote: On Wed, Jan 20, 2016 at 7:22 PM, Brett Cannon wrote: On Wed, 20 Jan 2016 at 10:11 Yury Selivanov wrote: [..] "ma_extra" would also make it easier for us to extend dicts in the future. Why can't

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 2:09 PM, Maciej Fijalkowski wrote: > You don't free a version tag that's stored in the guard. You store the object and not id Ah, got it. Although for my current cache design it would be more memory efficient to use the dict itself to store its own unique id and tag, hence my

Re: [Python-Dev] PEP 509: Add a private version to dict

there is also the problem that you don't want it on all dicts. So having two extra words is more to pay than having extra objects (also, comparison is cheaper for guards) On Wed, Jan 20, 2016 at 8:23 PM, Yury Selivanov wrote: > > > On 2016-01-20 2:09 PM, Maciej

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 2:02 PM, Maciej Fijalkowski wrote: On Wed, Jan 20, 2016 at 8:00 PM, Yury Selivanov wrote: [..] Brett, you need two things - the ID of the dict and the version tag. What we do in pypy is we have a small object (called, surprisingly, VersionTag()) and we

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, Jan 20, 2016 at 8:08 PM, Yury Selivanov wrote: > > On 2016-01-20 2:02 PM, Maciej Fijalkowski wrote: >> >> On Wed, Jan 20, 2016 at 8:00 PM, Yury Selivanov >> wrote: >> > [..] Brett, you need two things - the ID of the dict and

Re: [Python-Dev] PEP 509: Add a private version to dict

On 11/01/16 16:49, Victor Stinner wrote: Hi, After a first round on python-ideas, here is the second version of my PEP. The main changes since the first version are that the dictionary version is no more exposed at the Python level and the field type now also has a size of 64-bit on 32-bit

Re: [Python-Dev] PEP 509: Add a private version to dict

On 1/20/2016 12:50 PM, Brett Cannon wrote: A global (shared between all dicts) unit64 ma_version is actually quite reliable -- if a program does 1,000,000 dict modifications per second, it would take it 600,000 years till wrap-around. But would invalidate everything, instead

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, 20 Jan 2016 at 10:46 Yury Selivanov wrote: > Brett, > > On 2016-01-20 1:22 PM, Brett Cannon wrote: > > > > > > On Wed, 20 Jan 2016 at 10:11 Yury Selivanov > > wrote: > > > > On 2016-01-18 5:43 PM,

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 2:45 PM, Glenn Linderman wrote: For the reuse case, can't you simply keep the ma_version "live" in dict items on the free list, rather than starting over at (presumably) 0 ? Then if the dict is reused, it bumps the ma_version, and the fallback code goes on with (presumably)

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, 20 Jan 2016 at 12:27 Yury Selivanov wrote: > > > On 2016-01-20 2:45 PM, Glenn Linderman wrote: > > For the reuse case, can't you simply keep the ma_version "live" in > > dict items on the free list, rather than starting over at (presumably) > > 0 ? Then if the

Re: [Python-Dev] PEP 509: Add a private version to dict

Yury Selivanov wrote: What I propose is to add a pointer "ma_extra" (same 64bits), which will be set to NULL for most dict instances (instead of ma_version). "ma_extra" can then point to a struct that has a globally unique dict ID (uint64), and a version tag (unit64). Why not just use a

Re: [Python-Dev] PEP 509: Add a private version to dict

On 2016-01-20 1:15 PM, Maciej Fijalkowski wrote: [..] Anyway, I would strongly suggest getting some benchmarks done and showing performance benefits first, because you don't want PEPs to be final when you don't exactly know the details. I agree 100%. Yury

Re: [Python-Dev] PEP 509: Add a private version to dict

Hi, 2016-01-20 22:18 GMT+01:00 Glenn Linderman : > On 1/20/2016 12:50 PM, Brett Cannon wrote: >> >> A global (shared between all dicts) unit64 ma_version is actually quite >> reliable -- if a program does 1,000,000 dict modifications per second, >> it would take it 600,000

Re: [Python-Dev] PEP 509: Add a private version to dict

On Wed, 20 Jan 2016 at 15:46 Victor Stinner wrote: > Hi, > > 2016-01-20 22:18 GMT+01:00 Glenn Linderman : > > On 1/20/2016 12:50 PM, Brett Cannon wrote: > >> > >> A global (shared between all dicts) unit64 ma_version is actually quite > >>

Re: [Python-Dev] PEP 509: Add a private version to dict

On 1/20/2016 4:08 PM, Brett Cannon wrote: On Wed, 20 Jan 2016 at 15:46 Victor Stinner > wrote: Hi, 2016-01-20 22:18 GMT+01:00 Glenn Linderman >: > On 1/20/2016

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-21 1:08 GMT+01:00 Brett Cannon : > On Wed, 20 Jan 2016 at 15:46 Victor Stinner >> The worst case is when a value different than the watched value is >> modified between each guard check. In this case, we always need a dict >> lookup. An

Re: [Python-Dev] PEP 509: Add a private version to dict

Is someone opposed to this PEP 509? The main complain was the change on the public Python API, but the PEP doesn't change the Python API anymore. I'm not aware of any remaining issue on this PEP. Victor 2016-01-11 17:49 GMT+01:00 Victor Stinner : > Hi, > > After a

Re: [Python-Dev] PEP 509: Add a private version to dict

On Jan 18, 2016, at 11:43 PM, Victor Stinner wrote: >Is someone opposed to this PEP 509? > >The main complain was the change on the public Python API, but the PEP >doesn't change the Python API anymore. > >I'm not aware of any remaining issue on this PEP. Have you done any performance analysis

Re: [Python-Dev] PEP 509: Add a private version to dict

Well, it was just a remark. 2016-01-12 0:35 GMT+01:00 Andrew Barnert : > Are you saying that d[key] = d[key] may or may not increment the version, so > any optimizer can't rely on the fact that it doesn't? Optimizers don't have to rely on this exactly behaviour. Not

Re: [Python-Dev] PEP 509: Add a private version to dict

On Mon, Jan 11, 2016 at 8:50 AM Victor Stinner wrote: > Hi, > > After a first round on python-ideas, here is the second version of my > PEP. The main changes since the first version are that the dictionary > version is no more exposed at the Python level and the field

Re: [Python-Dev] PEP 509: Add a private version to dict

2016-01-12 0:07 GMT+01:00 Gregory P. Smith : >> Changes >> === >> >> (...) > > Please be more explicit about what tests you are performing on the values. > setitem's "if the value is different" really should mean "if value is not > dict['key']". similarly for update, there

Re: [Python-Dev] PEP 509: Add a private version to dict

On Jan 11, 2016, at 15:24, Victor Stinner wrote: > > 2016-01-12 0:07 GMT+01:00 Gregory P. Smith : >>> Changes >>> === >>> >>> (...) >> >> Please be more explicit about what tests you are performing on the values. >> setitem's "if the value is

[Python-Dev] PEP 509: Add a private version to dict

Hi, After a first round on python-ideas, here is the second version of my PEP. The main changes since the first version are that the dictionary version is no more exposed at the Python level and the field type now also has a size of 64-bit on 32-bit platforms. The PEP is part of a serie of 3 PEP

Re: [Python-Dev] PEP 509: Add a private version to dict

Hi Victor. You know that pypy does this stuff without changing and exposing python semantics right? We have a version dict that does not leak abstractions to the user. In general, doing stuff like that where there is a public API that leaks details of certain optimizations makes it harder and

Re: [Python-Dev] PEP 509: Add a private version to dict

Le 11 janv. 2016 8:09 PM, "Maciej Fijalkowski" a écrit : > Hi Victor. > > You know that pypy does this stuff without changing and exposing > python semantics right? We have a version dict that does not leak > abstractions to the user. The PEP adds a field to the C structure

Re: [Python-Dev] PEP 509: Add a private version to dict

On Mon, Jan 11, 2016 at 9:56 PM, Victor Stinner wrote: > Le 11 janv. 2016 8:09 PM, "Maciej Fijalkowski" a écrit : >> Hi Victor. >> >> You know that pypy does this stuff without changing and exposing >> python semantics right? We have a version dict