subject:"Re\: \[Python\-Dev\] PEP 455\: TransformDict"

Re: [Python-Dev] PEP 455 -- TransformDict

2015-05-14 Thread Guido van Rossum

Thanks for this thorough review, Raymond! Especially the user research is
amazing.

 And thanks for Antoine for writing the PEP -- you never know how an idea
pans out until you've tried it.

--Guido

On Thu, May 14, 2015 at 7:29 AM, Raymond Hettinger 
raymond.hettin...@gmail.com wrote:

 Before the Python 3.5 feature freeze, I should step-up and
 formally reject PEP 455 for Adding a key-transforming
 dictionary to collections.

 I had completed an involved review effort a long time ago
 and I apologize for the delay in making the pronouncement.

 What made it a interesting choice from the outset is that the
 idea of a transformation is an enticing concept that seems
 full of possibility.  I spent a good deal of time exploring
 what could be done with it but found that it mostly fell short
 of its promise.

 There were many issues.  Here are some that were at the top:

 * Most use cases don't need or want the reverse lookup feature
   (what is wanted is a set of one-way canonicalization functions).
   Those that do would want to have a choice of what is saved
   (first stored, last stored, n most recent, a set of all inputs,
   a list of all inputs, nothing, etc).  In database terms, it
   models a many-to-one table (the canonicalization or
   transformation function) with the one being a primary key into
   another possibly surjective table of two columns (the
   key/value store).  A surjection into another surjection isn't
   inherently reversible in a useful way, nor does it seem to be a
   common way to model data.

 * People are creative at coming up with using cases for the TD
   but then find that the resulting code is less clear, slower,
   less intuitive, more memory intensive, and harder to debug than
   just using a plain dict with a function call before the lookup:
   d[func(key)].  It was challenging to find any existing code
   that would be made better by the availability of the TD.

 * The TD seems to be all about combining data scrubbing
   (case-folding, unicode canonicalization, type-folding, object
   identity, unit-conversion, or finding a canonical member of an
   equivalence class) with a mapping (looking-up a value for a
   given key).  Those two operations are conceptually orthogonal.
   The former doesn't get easier when hidden behind a mapping API
   and the latter loses the flexibility of choosing your preferred
   mapping (an ordereddict, a persistentdict, a chainmap, etc) and
   the flexibility of establishing your own rules for whether and
   how to do a reverse lookup.


 Raymond Hettinger


 P.S.  Besides the core conceptual issues listed above, there
 are a number of smaller issues with the TD that surfaced
 during design review sessions.  In no particular order, here
 are a few of the observations:

 * It seems to require above average skill to figure-out what
   can be used as a transform function.  It is more
   expert-friendly than beginner friendly.  It takes a little
   while to get used to it.  It wasn't self-evident that
   transformations happen both when a key is stored and again
   when it is looked-up (contrast this with key-functions for
   sorting which are called at most once per key).

 * The name, TransformDict, suggests that it might transform the
   value instead of the key or that it might transform the
   dictionary into something else.  The name TransformDict is so
   general that it would be hard to discover when faced with a
   specific problem.  The name also limits perception of what
   could be done with it (i.e. a function that logs accesses
   but doesn't actually change the key).

 * The tool doesn't self describe itself well.  Looking at the
   help(), or the __repr__(), or the tooltips did not provide
   much insight or clarity.  The dir() shows many of the
   _abc implementation details rather than the API itself.

 * The original key is stored and if you change it, the change
   isn't stored.  The _original dict is private (perhaps to
   reduce the risk of putting the TD in an inconsistent state)
   but this limits access to the stored data.

 * The TD is unsuitable for bijections because the API is
   inherently biased with a rich group of operators and methods
   for forward lookup but has only one method for reverse lookup.

 * The reverse feature is hard to find (getitem vs __getitem__)
   and its output pair is surprising and a bit awkward to use.
   It provides only one accessor method rather that the full
   dict API that would be given by a second dictionary.  The
   API hides the fact that there are two underlying dictionaries.

 * It was surprising that when d[k] failed, it failed with
   transformation exception rather than a KeyError, violating
   the expectations of the calling code (for example, if the
   transformation function is int(), the call d[12]
   transforms to d[12] and either succeeds in returning a value
   or in raising a KeyError, but the call d[12.0] fails with
   a TypeError).  The latter issue limits its

Re: [Python-Dev] PEP 455 -- TransformDict

2015-05-14 Thread Nick Coghlan

On 15 May 2015 at 00:41, Guido van Rossum gu...@python.org wrote:
 Thanks for this thorough review, Raymond! Especially the user research is
 amazing.

  And thanks for Antoine for writing the PEP -- you never know how an idea
 pans out until you've tried it.

Hear, hear! I thought the TransformDict idea sounded interesting when
Antoine proposed it, but Raymond's rationale for the rejection makes a
great deal of sense.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 455: TransformDict

2013-10-30 Thread Raymond Hettinger


On Oct 28, 2013, at 1:16 PM, Victor Stinner victor.stin...@gmail.com wrote:

 so what is the
 status of the PEP 455 (TransformDict)?


I'm giving a thorough evaluation of the proposal
and am devoting chunks of time each weekend
to reviewing the email threads, the links provided
in the PEPs, looking at how well the TD fits in existing code.

I'm giving this work priority over my own list of things
to add to 3.4 (most of which will now have to wait until 3.5).

This week, I'm teaching a five-day intermediate python class
to highly experienced network engineers in Denver.  We'll do
some exercises using the TD and evaluate the results against
alternative approaches.

Here are some preliminary notes (in no particular order):

* A first reading of the python-dev threads suggests that
the two-dict TD implementation seems better suited to
implementing a case-folding-case-preserving dictionary
and is far less well suited for a case-folding dict or an
identity dict.

* There are interesting differences between the proposed TD
and the CaseInsensitiveDict implemented in Kenneth Reitz's
HTTP requests library.  The latter keeps the last key added
rather than the first.   It also has a cleaner implementation
and the API is a bit nicer (no getitem() method).

* The originals dict maps a transformed key back to its
first saved original value.  An alternative would be to map
back to a set of original values or a list of original values.

* A possible use case is for a unicode normalizing dictionary
where  'L' + chr(111) + chr(776) + 'wis'  would match
'L' + chr(246) + 'wis'.

* The implementation looks rough at this point, but that is easily
fixed-up.  I'll leave some specific suggestions on the tracker
(getting it to accept a list of tuples in the constructor, a recursive
repr, renaming the getitem() method, deprivatizing the attributes,
getting it to work with __missing__, etc).

* Having two-mappings-in-one seems to be the most awkward
part of the design and wouldn't be needed for the two strongest
use cases, a case-insensitive-but-not-case-preserving dict
and an identity dict.

* In http://stackoverflow.com/questions/13230414, the OP
wants a CI dict but doesn't need the case preserving aspect.
The OP is provided with a regular dictionary containing mixed case
keys and needs to compare a list of potential matches of unknown case. 
Copying the dict to a case-folding TD wouldn't provide any incremental
advantage over building a regular dict with lower case keys.

* In http://www.gossamer-threads.com/lists/python/python/209527,
the OP wanted an ID comparison for a symbolic calculation.
The objects already had an eq function and he wanted to temporarily
bypass that in a symbol lookup.  Essentially he needed a dictionary
that would allow the use of an alternative equality function.
A TD would work here but there would be no need for the 
key preserving feature.  There doesn't seem to be any advantage
over using a regular dict that directly stores the id() as the key.

* In https://mail.python.org/pipermail/python-ideas/2010-May/007235.html,
the OP wants a highly optimized identity dictionary that doesn't
call an expensive id() function.   The proposed TD doesn't fulfill
this request -- it would be far from being optimized and would
call id() on every store and every lookup.

* In http://msdn.microsoft.com/en-us/library/xfhwa508.aspx,
the API describes a dict with an alternative equality comparison.
This is a different design than the TD and is for a world that is
somewhat different from Python.  In that dict, eq and hash
are specified at the dict level rather than object level
(in Python, the key objects define their own __eq__ and __hash__
rather than having the user attach the functions directly to the dictionary).

* In http://docs.oracle.com/javase/6/docs/api/java/util/IdentityHashMap.html,
the IdentityHashMap is described as being for rare use cases
such as topology-preserving object graph transformations
likes serialization or deep-copying.  Looking at Python's own code
for copy.deepcopy(), the TD would not be a good replacement
for the existing code (more memory intensive, slower, no use
for the originals dict, and no use for most of the TD functionality).
It appears that it is simpler and faster to store and lookup d[id(obj)] than
to use a TD.

* If this were being modeled in a database, we would have one table
with a many-to-one mapping of original keys to transformed keys
and another table with a transformed key as the primary key in a
table of key-value pairs.   This suggests two lookup patterns
original-tranformed-value and transformed-all_originals.

* The Apache case insensitive dict documentation includes these
thoughts: This map will violate the detail of various Map and map view 
contracts. As a
general rule, don't compare this map to other maps. In particular, you can't use
decorators like ListOrderedMap on it, which silently assume that these contracts
are fulfilled. --- Note that CaseInsensitiveMap is

Re: [Python-Dev] PEP 455: TransformDict

2013-10-30 Thread R. David Murray

On Wed, 30 Oct 2013 01:12:03 -0600, Raymond Hettinger 
raymond.hettin...@gmail.com wrote:
 If I had to choose right now, a safe choice would be to focus on
 the primary use case and implement a clean CaseInsensitiveDict
 without the double-dict first-saved case-preserving feature.
 That said, I find the TD to be fascinating and there's more work
 to do before making a decision.

Please be aware that the PEP author's motivation in submitting the PEP was
to have a case insensitive, case *preserving* dict.  The generalization
serves to make the new datatype more useful, but if the end result
doesn't satisfy the original use case of the author, I won't be
surprised if he has no motivation to work on it further :).

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 455: TransformDict

2013-10-30 Thread R. David Murray

On Wed, 30 Oct 2013 01:12:03 -0600, Raymond Hettinger 
raymond.hettin...@gmail.com wrote:
 I'm giving a thorough evaluation of the proposal
 and am devoting chunks of time each weekend
 to reviewing the email threads, the links provided
 in the PEPs, looking at how well the TD fits in existing code.
 
 I'm giving this work priority over my own list of things
 to add to 3.4 (most of which will now have to wait until 3.5).

And thanks for doing all this work, Raymond.  I forgot to say
that in my previous post.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 455: TransformDict

2013-10-30 Thread Ethan Furman


On 10/30/2013 12:12 AM, Raymond Hettinger wrote:


Hopefully, this post will make the thought process more transparent.


Thanks, Raymond.  Your time is appreciated.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 455: TransformDict

2013-10-30 Thread Nigel Small

It strikes me that there could be an alternative approach to some of the
use cases discussed here. Instead of a new type of dictionary, the
case-insensitivity problem could be solved with something akin to a *
CaseInsensitiveString* class used for keys within a standard dictionary.
This would be very similar to a normal string except with comparison and
hashing. It would mean that CaseInsensitiveString(Foo) is considered
equal to CaseInsensitiveString(foo) and allow code such as the following:

headers = {}
headers[CaseInsensitiveString(content-type)] = text/plain
headers[CaseInsensitiveString(Content-Type)]
text/plain

This would obviously also be usable in other places where case-insensitive
strings are required.

Just my two pence/cents/other minor currency units.
Nigel

On 30 October 2013 14:18, Ethan Furman et...@stoneleaf.us wrote:

On 10/30/2013 12:12 AM, Raymond Hettinger wrote:

Hopefully, this post will make the thought process more transparent.

Thanks, Raymond. Your time is appreciated.

--
~Ethan~

__**_
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/**mailman/listinfo/python-devhttps://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/**mailman/options/python-dev/**
nigel%40nigelsmall.comhttps://mail.python.org/mailman/options/python-dev/nigel%40nigelsmall.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

1 2 >

1 - 100 of 101 matches

Mail list logo