Re: Overriding iadd for dictionary like objects
In article <7f82416a-53be-41b3-9503-1492454cc...@upsg2000gro.googlegroups.com>,
RunThePun wrote:
>On Sep 1, 3:00=A0am, a...@pythoncraft.com (Aahz) wrote:
>> In article .com>,
>> RunThePun =A0 wrote:
>>>On Aug 30, 10:33=3DA0pm, a...@pythoncraft.com (Aahz) wrote:
In article ups=3D
>>>.com>,
RunThePun =3DA0 wrote:
>>
>I made a DictMixin where the keys are filenames and the values are the
>file contents. It was very simple and easy to do thanks to DictMixin.
>>
>For example this code writes "abc" in a file named "temp.txt" and
>prints the contents of the file named "swallow", these files are
>looked up/created/deleted in the directory "spam":
d =3D3D3D FilesDict('spam')
d['temp.txt'] =3D3D3D 'abc'
print(d['swallow'])
>>
>My problem arose when I wanted to append a string to a file which
>using open(..., 'ab') would have been miles more efficient because I
>wouldn't have to read the entire file (__getitem__) and then write the
>entire file back (__setitem__). The files are expected to be as big as
>600 KB which will be appended 30 bytes at a time about 3 times a
>second. Performance-wise the system would probably work without open
>(..., 'ab') but it would be a real thrashing so the current solution
>uses a method "AddTo" as Robert suggested, sacrificing the neat
>getitem/setitem syntax.
>>
You can do mostly what you want, I think, by having __setitem__()
convert string values into FileProxy() objects that have an appropriat=
>e
__iadd__() method. =3DA0That brings a whole new set of problems, of co=
>urse.
>>
>>>I'm guessing you meant __getitem__, which is what Jan Kaliszewski
>>>suggested, but as you noted, would be a bit cumbersome in this case.
>>
>> Actually, what I meant was __setitem__. =A0The idea is that you create th=
>e
>> proxy item when you add the data to the dict (wrapping the original
>> data), and the proxy has an __iadd__ method, which would allow you to do
>> the file append.
>
>But you do mean that __getitem__ would return a wrapped object as
>well, right? Otherwise I don't see how the iadd would be relevant
>because:
> d['a'] +=3D 3
>is equivalent to:
> d.__setitem__('a', d.__getitem__('a').__iadd__(3))
The __getitem__ returns the proxy object created by the first __setitem__
when you did
d['a'] = 'something'
The __iadd__ of the proxy object returns self, so it's the same object,
which means that the second __setitem__ (called by +=) can check to see
that you already have a proxy object and it does not need to rewrap it.
If this still doesn't make sense, I suggest you go ahead and try the
experiment to prove that it does work. ;-)
--
Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/
"Look, it's your affair if you want to play with five people, but don't
go calling it doubles." --John Cleese anticipates Usenet
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Sep 1, 3:00 am, a...@pythoncraft.com (Aahz) wrote:
> In article
> ,
>
>
>
>
>
> RunThePun wrote:
> >On Aug 30, 10:33=A0pm, a...@pythoncraft.com (Aahz) wrote:
> >> In article >.com>,
> >> RunThePun =A0 wrote:
>
> >>>I made a DictMixin where the keys are filenames and the values are the
> >>>file contents. It was very simple and easy to do thanks to DictMixin.
>
> >>>For example this code writes "abc" in a file named "temp.txt" and
> >>>prints the contents of the file named "swallow", these files are
> >>>looked up/created/deleted in the directory "spam":
> >> d =3D3D FilesDict('spam')
> >> d['temp.txt'] =3D3D 'abc'
> >> print(d['swallow'])
>
> >>>My problem arose when I wanted to append a string to a file which
> >>>using open(..., 'ab') would have been miles more efficient because I
> >>>wouldn't have to read the entire file (__getitem__) and then write the
> >>>entire file back (__setitem__). The files are expected to be as big as
> >>>600 KB which will be appended 30 bytes at a time about 3 times a
> >>>second. Performance-wise the system would probably work without open
> >>>(..., 'ab') but it would be a real thrashing so the current solution
> >>>uses a method "AddTo" as Robert suggested, sacrificing the neat
> >>>getitem/setitem syntax.
>
> >> You can do mostly what you want, I think, by having __setitem__()
> >> convert string values into FileProxy() objects that have an appropriate
> >> __iadd__() method. =A0That brings a whole new set of problems, of course.
>
> >I'm guessing you meant __getitem__, which is what Jan Kaliszewski
> >suggested, but as you noted, would be a bit cumbersome in this case.
>
> Actually, what I meant was __setitem__. The idea is that you create the
> proxy item when you add the data to the dict (wrapping the original
> data), and the proxy has an __iadd__ method, which would allow you to do
> the file append.
> --
> Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/
>
> "I support family values -- Addams family values" --www.nancybuttons.com
But you do mean that __getitem__ would return a wrapped object as
well, right? Otherwise I don't see how the iadd would be relevant
because:
d['a'] += 3
is equivalent to:
d.__setitem__('a', d.__getitem__('a').__iadd__(3))
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
In article ,
RunThePun wrote:
>On Aug 30, 10:33=A0pm, a...@pythoncraft.com (Aahz) wrote:
>> In article .com>,
>> RunThePun =A0 wrote:
>>>
>>>I made a DictMixin where the keys are filenames and the values are the
>>>file contents. It was very simple and easy to do thanks to DictMixin.
>>>
>>>For example this code writes "abc" in a file named "temp.txt" and
>>>prints the contents of the file named "swallow", these files are
>>>looked up/created/deleted in the directory "spam":
>> d =3D3D FilesDict('spam')
>> d['temp.txt'] =3D3D 'abc'
>> print(d['swallow'])
>>>
>>>My problem arose when I wanted to append a string to a file which
>>>using open(..., 'ab') would have been miles more efficient because I
>>>wouldn't have to read the entire file (__getitem__) and then write the
>>>entire file back (__setitem__). The files are expected to be as big as
>>>600 KB which will be appended 30 bytes at a time about 3 times a
>>>second. Performance-wise the system would probably work without open
>>>(..., 'ab') but it would be a real thrashing so the current solution
>>>uses a method "AddTo" as Robert suggested, sacrificing the neat
>>>getitem/setitem syntax.
>>
>> You can do mostly what you want, I think, by having __setitem__()
>> convert string values into FileProxy() objects that have an appropriate
>> __iadd__() method. =A0That brings a whole new set of problems, of course.
>
>I'm guessing you meant __getitem__, which is what Jan Kaliszewski
>suggested, but as you noted, would be a bit cumbersome in this case.
Actually, what I meant was __setitem__. The idea is that you create the
proxy item when you add the data to the dict (wrapping the original
data), and the proxy has an __iadd__ method, which would allow you to do
the file append.
--
Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/
"I support family values -- Addams family values" --www.nancybuttons.com
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Aug 30, 10:33 pm, a...@pythoncraft.com (Aahz) wrote:
> In article
> ,
>
>
>
>
>
> RunThePun wrote:
>
> >I made a DictMixin where the keys are filenames and the values are the
> >file contents. It was very simple and easy to do thanks to DictMixin.
>
> >For example this code writes "abc" in a file named "temp.txt" and
> >prints the contents of the file named "swallow", these files are
> >looked up/created/deleted in the directory "spam":
> d =3D FilesDict('spam')
> d['temp.txt'] =3D 'abc'
> print(d['swallow'])
>
> >My problem arose when I wanted to append a string to a file which
> >using open(..., 'ab') would have been miles more efficient because I
> >wouldn't have to read the entire file (__getitem__) and then write the
> >entire file back (__setitem__). The files are expected to be as big as
> >600 KB which will be appended 30 bytes at a time about 3 times a
> >second. Performance-wise the system would probably work without open
> >(..., 'ab') but it would be a real thrashing so the current solution
> >uses a method "AddTo" as Robert suggested, sacrificing the neat
> >getitem/setitem syntax.
>
> You can do mostly what you want, I think, by having __setitem__()
> convert string values into FileProxy() objects that have an appropriate
> __iadd__() method. That brings a whole new set of problems, of course.
> --
> Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/
>
> "I support family values -- Addams family values" --www.nancybuttons.com
I'm guessing you meant __getitem__, which is what Jan Kaliszewski
suggested, but as you noted, would be a bit cumbersome in this case.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
In article ,
RunThePun wrote:
>
>I made a DictMixin where the keys are filenames and the values are the
>file contents. It was very simple and easy to do thanks to DictMixin.
>
>For example this code writes "abc" in a file named "temp.txt" and
>prints the contents of the file named "swallow", these files are
>looked up/created/deleted in the directory "spam":
d =3D FilesDict('spam')
d['temp.txt'] =3D 'abc'
print(d['swallow'])
>
>My problem arose when I wanted to append a string to a file which
>using open(..., 'ab') would have been miles more efficient because I
>wouldn't have to read the entire file (__getitem__) and then write the
>entire file back (__setitem__). The files are expected to be as big as
>600 KB which will be appended 30 bytes at a time about 3 times a
>second. Performance-wise the system would probably work without open
>(..., 'ab') but it would be a real thrashing so the current solution
>uses a method "AddTo" as Robert suggested, sacrificing the neat
>getitem/setitem syntax.
You can do mostly what you want, I think, by having __setitem__()
convert string values into FileProxy() objects that have an appropriate
__iadd__() method. That brings a whole new set of problems, of course.
--
Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/
"I support family values -- Addams family values" --www.nancybuttons.com
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
PS. Sorry for sending 2 posts -- the latter is the correct one. Cheers, *j -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Aug 29, 1:58 pm, Carl Banks wrote:
> On Aug 28, 10:37 pm, Joshua Judson Rosen wrote:
>
>
>
>
>
> > Carl Banks writes:
>
> > > On Aug 28, 2:42 pm, Terry Reedy wrote:
>
> > > > Carl Banks wrote:
> > > > > I don't think it needs a syntax for that, but I'm not so sure a method
> > > > > to modify a value in place with a single key lookup wouldn't
> > > > > occasioanally be useful.
>
> > > > Augmented assignment does that.
>
> > > Internally uses two lookups, one for getting, and one for setting.
>
> > > I think this is an unavoidable given Python's semantics. Look at
> > > the traceback:
>
> > > >>> def x():
> > > ... d['a'] += 1
> > > ...
> > > >>> dis.dis(x)
> > > 2 0 LOAD_GLOBAL 0 (d)
> > > 3 LOAD_CONST 1 ('a')
> > > 6 DUP_TOPX 2
> > > 9 BINARY_SUBSCR
>
> > OK, there's one lookup, but...
>
> > > 10 LOAD_CONST 2 (1)
> > > 13 INPLACE_ADD
> > > 14 ROT_THREE
> > > 15 STORE_SUBSCR
> > > 16 LOAD_CONST 0 (None)
> > > 19 RETURN_VALUE
>
> > ... I don't see anything in there that retrieves the value a second time
>
> STORE_SUBSCR has to look up the position in the hash table to store
> the value, hence the second lookup.
>
> > > > > As a workaround, if lookups are expensive,
>
> > > > But they are not. Because (C)Python is heavily based on dict name lookup
> > > > for builtins and global names and attributes, as well as overt dict
> > > > lookup, must effort has gone into optimizing dict lookup.
>
> > > The actual lookup algorithm Python dicts use is well-optimized, yes,
> > > but the dict could contain keys that have expensive comparison and
> > > hash-code calculation, in which case lookup is going to be slow.
>
> > I'll like the originator correct me if I've made a mistake, but I read
> > "lookup" as actually meaning "lookup", not "value-comparison".
>
> This has nothing to do with value comparison. I was talking about key
> comparison, which happens when looking up a position in a hash table.
> I was the first person to use the word "lookup" in this thread and I
> specifically meant hash-table position lookup.
>
> > At least in part because the question, as it was posed, specifically
> > related to a wrapper-class (providing a mapping ("dict like") interface)
> > around a database of some sort other than Python's dict class per se.
>
> > How do the details of Python's native dict-type's internal (hashtable)
> > algorithm matter when they're explicitly /not/ being used?
>
> Well it doesn't apply specifically to the OP's problem. I changed the
> topic a bit by making it specific to dicts. Is that ok with you? Was
> that not allowed?
>
> The OP can add a method like apply_to_value to his own class, but one
> can't do that for dicts. Ergo why something like apply_to_value()
> would be useful enough in rare circumstances where lookup is very slow
> to merit a moments consideration before being rejected.
>
> (If dict did have a method like that, the OP would at least know which
> method to override.)
>
> Carl Banks
First of all I'd like to say thanks for this discussion, you guys are
awesome.
I probably should have explained my problem better to begin with and I
apologize for that. So now I'll start from the top:
I made a DictMixin where the keys are filenames and the values are the
file contents. It was very simple and easy to do thanks to DictMixin.
For example this code writes "abc" in a file named "temp.txt" and
prints the contents of the file named "swallow", these files are
looked up/created/deleted in the directory "spam":
>>> d = FilesDict('spam')
>>> d['temp.txt'] = 'abc'
>>> print(d['swallow'])
This was very convenient for me because I wanted to use a simple DB
which could be read and edited by shell scripts and non-pythonistas,
without the heavy ORM. Also, if in the future an online full featured
DB would be needed, I could easily convert the DictMixin methods. So
up to here I had a good solution.
My problem arose when I wanted to append a string to a file which
using open(..., 'ab') would have been miles more efficient because I
wouldn't have to read the entire file (__getitem__) and then write the
entire file back (__setitem__). The files are expected to be as big as
600 KB which will be appended 30 bytes at a time about 3 times a
second. Performance-wise the system would probably work without open
(..., 'ab') but it would be a real thrashing so the current solution
uses a method "AddTo" as Robert suggested, sacrificing the neat
getitem/setitem syntax.
Just so I don't leave out any information, I actually also made a
DirectoryDict for having multiple 'tables'. In this DictMixin, keys
are directory names and values are FilesDict instances. So to write
"European or African" to the file "/root/tmp/velocity" one would be
just:
>>> d = DirectoryDict("/root")
>>> d["tmp"]["velocity"]
Re: Overriding iadd for dictionary like objects
On Aug 28, 10:37 pm, Joshua Judson Rosen wrote:
> Carl Banks writes:
>
> > On Aug 28, 2:42 pm, Terry Reedy wrote:
>
> > > Carl Banks wrote:
> > > > I don't think it needs a syntax for that, but I'm not so sure a method
> > > > to modify a value in place with a single key lookup wouldn't
> > > > occasioanally be useful.
>
> > > Augmented assignment does that.
>
> > Internally uses two lookups, one for getting, and one for setting.
>
> > I think this is an unavoidable given Python's semantics. Look at
> > the traceback:
>
> > >>> def x():
> > ... d['a'] += 1
> > ...
> > >>> dis.dis(x)
> > 2 0 LOAD_GLOBAL 0 (d)
> > 3 LOAD_CONST 1 ('a')
> > 6 DUP_TOPX 2
> > 9 BINARY_SUBSCR
>
> OK, there's one lookup, but...
>
> > 10 LOAD_CONST 2 (1)
> > 13 INPLACE_ADD
> > 14 ROT_THREE
> > 15 STORE_SUBSCR
> > 16 LOAD_CONST 0 (None)
> > 19 RETURN_VALUE
>
> ... I don't see anything in there that retrieves the value a second time
STORE_SUBSCR has to look up the position in the hash table to store
the value, hence the second lookup.
> > > > As a workaround, if lookups are expensive,
>
> > > But they are not. Because (C)Python is heavily based on dict name lookup
> > > for builtins and global names and attributes, as well as overt dict
> > > lookup, must effort has gone into optimizing dict lookup.
>
> > The actual lookup algorithm Python dicts use is well-optimized, yes,
> > but the dict could contain keys that have expensive comparison and
> > hash-code calculation, in which case lookup is going to be slow.
>
> I'll like the originator correct me if I've made a mistake, but I read
> "lookup" as actually meaning "lookup", not "value-comparison".
This has nothing to do with value comparison. I was talking about key
comparison, which happens when looking up a position in a hash table.
I was the first person to use the word "lookup" in this thread and I
specifically meant hash-table position lookup.
> At least in part because the question, as it was posed, specifically
> related to a wrapper-class (providing a mapping ("dict like") interface)
> around a database of some sort other than Python's dict class per se.
>
> How do the details of Python's native dict-type's internal (hashtable)
> algorithm matter when they're explicitly /not/ being used?
Well it doesn't apply specifically to the OP's problem. I changed the
topic a bit by making it specific to dicts. Is that ok with you? Was
that not allowed?
The OP can add a method like apply_to_value to his own class, but one
can't do that for dicts. Ergo why something like apply_to_value()
would be useful enough in rare circumstances where lookup is very slow
to merit a moments consideration before being rejected.
(If dict did have a method like that, the OP would at least know which
method to override.)
Carl Banks
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Sat, 29 Aug 2009 01:23:59 -0400, Joshua Judson Rosen wrote: > Robert Kern writes: >> >> On 2009-08-28 16:42 PM, Terry Reedy wrote: >> > Carl Banks wrote: >> > >> > > I don't think it needs a syntax for that, but I'm not so sure a >> > > method to modify a value in place with a single key lookup wouldn't >> > > occasioanally be useful. >> > >> > Augmented assignment does that. >> >> No, it uses one __getitem__ and one __setitem__ thus two key lookups. > > Apparently you're defining "key lookup" some other way than as `what > __getitem__ does'. > > What exactly does "key lookup" mean to you? > > I've always understood it as `retrieving the value associated with a > key', which obviously isn't required for assignment--otherwise it > wouldn't be possible to add new keys to a mapping. When you retrieve a value from a dictionary using __getitem__, e.g.: dict["K"] the dict has to search the hash table for the record with key "K". This a key lookup. (I use the term "search", but of course for hash tables this is usually very fast. For Python dicts, you can assume it will usually be a constant time, independent of the key or the size of the dict.) When you store a value in a dictionary using __setitem__, e.g.: dict["K"] = 42 the dict has to search the hash table for the correct place to store a record with key "K". It obviously can't place the record in some arbitrary place, it has to be in the correct place for future lookups to find it. This is also a key lookup. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
Carl Banks writes:
>
> On Aug 28, 2:42 pm, Terry Reedy wrote:
>
> > Carl Banks wrote:
> > > I don't think it needs a syntax for that, but I'm not so sure a method
> > > to modify a value in place with a single key lookup wouldn't
> > > occasioanally be useful.
> >
> > Augmented assignment does that.
>
> Internally uses two lookups, one for getting, and one for setting.
>
> I think this is an unavoidable given Python's semantics. Look at
> the traceback:
>
>
> >>> def x():
> ... d['a'] += 1
> ...
> >>> dis.dis(x)
> 2 0 LOAD_GLOBAL 0 (d)
> 3 LOAD_CONST 1 ('a')
> 6 DUP_TOPX 2
> 9 BINARY_SUBSCR
OK, there's one lookup, but...
> 10 LOAD_CONST 2 (1)
> 13 INPLACE_ADD
> 14 ROT_THREE
> 15 STORE_SUBSCR
> 16 LOAD_CONST 0 (None)
> 19 RETURN_VALUE
... I don't see anything in there that retrieves the value a second time
> > > As a workaround, if lookups are expensive,
> >
> > But they are not. Because (C)Python is heavily based on dict name lookup
> > for builtins and global names and attributes, as well as overt dict
> > lookup, must effort has gone into optimizing dict lookup.
>
> The actual lookup algorithm Python dicts use is well-optimized, yes,
> but the dict could contain keys that have expensive comparison and
> hash-code calculation, in which case lookup is going to be slow.
I'll like the originator correct me if I've made a mistake, but I read
"lookup" as actually meaning "lookup", not "value-comparison".
At least in part because the question, as it was posed, specifically
related to a wrapper-class (providing a mapping ("dict like") interface)
around a database of some sort other than Python's dict class per se.
How do the details of Python's native dict-type's internal (hashtable)
algorithm matter when they're explicitly /not/ being used?
--
Don't be afraid to ask (Lf.((Lx.xx) (Lr.f(rr.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
Robert Kern writes: > > On 2009-08-28 16:42 PM, Terry Reedy wrote: > > Carl Banks wrote: > > > > > I don't think it needs a syntax for that, but I'm not so sure a > > > method to modify a value in place with a single key lookup > > > wouldn't occasioanally be useful. > > > > Augmented assignment does that. > > No, it uses one __getitem__ and one __setitem__ thus two key lookups. Apparently you're defining "key lookup" some other way than as `what __getitem__ does'. What exactly does "key lookup" mean to you? I've always understood it as `retrieving the value associated with a key', which obviously isn't required for assignment--otherwise it wouldn't be possible to add new keys to a mapping. -- Don't be afraid to ask (Lf.((Lx.xx) (Lr.f(rr. -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Aug 28, 2:42 pm, Terry Reedy wrote:
> Carl Banks wrote:
> > I don't think it needs a syntax for that, but I'm not so sure a method
> > to modify a value in place with a single key lookup wouldn't
> > occasioanally be useful.
>
> Augmented assignment does that.
Internally uses two lookups, one for getting, and one for setting. I
think this is an unavoidable given Python's semantics. Look at the
traceback:
>>> def x():
... d['a'] += 1
...
>>> dis.dis(x)
2 0 LOAD_GLOBAL 0 (d)
3 LOAD_CONST 1 ('a')
6 DUP_TOPX 2
9 BINARY_SUBSCR
10 LOAD_CONST 2 (1)
13 INPLACE_ADD
14 ROT_THREE
15 STORE_SUBSCR
16 LOAD_CONST 0 (None)
19 RETURN_VALUE
> > As a workaround, if lookups are expensive,
>
> But they are not. Because (C)Python is heavily based on dict name lookup
> for builtins and global names and attributes, as well as overt dict
> lookup, must effort has gone into optimizing dict lookup.
The actual lookup algorithm Python dicts use is well-optimized, yes,
but the dict could contain keys that have expensive comparison and
hash-code calculation, in which case lookup is going to be slow.
Carl Banks
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On 2009-08-28 16:42 PM, Terry Reedy wrote:
Carl Banks wrote:
I don't think it needs a syntax for that, but I'm not so sure a method
to modify a value in place with a single key lookup wouldn't
occasioanally be useful.
Augmented assignment does that.
No, it uses one __getitem__ and one __setitem__ thus two key lookups.
For instance:
def increment(value):
return value+1
d = { 'a': 1 }
d.apply_to_value('a',increment)
print d
and d['a'] would be 2. The difference is that only one lookup
occurs.
Like this?
>>> d={'a': 2}
>>> d['a'] += 2
>>> d['a']
4
This does not cover all replacements, but certainly the most common.
Take look farther up in the thread for the actual point at issue. The OP knows
that augmented assignment works for the common cases. He wants to override the
behavior on the container to handle some uncommon cases, and this is not
possible because Python breaks up the operation into three orthogonal actions.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
Carl Banks wrote:
I don't think it needs a syntax for that, but I'm not so sure a method
to modify a value in place with a single key lookup wouldn't
occasioanally be useful.
Augmented assignment does that.
For instance:
def increment(value):
return value+1
d = { 'a': 1 }
d.apply_to_value('a',increment)
print d
and d['a'] would be 2. The difference is that only one lookup
occurs.
Like this?
>>> d={'a': 2}
>>> d['a'] += 2
>>> d['a']
4
This does not cover all replacements, but certainly the most common.
[snip]
As a workaround, if lookups are expensive,
But they are not. Because (C)Python is heavily based on dict name lookup
for builtins and global names and attributes, as well as overt dict
lookup, must effort has gone into optimizing dict lookup.
> you can add
something even slower ;-). In particular, a method lookup + method
call, as you suggest above.
One can always avoid calculating the key object twice if that is expensive.
Terry Jan Reedy
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Aug 26, 11:49 pm, RunThePun wrote:
> Anybody have any more ideas? I think python should/could havev a
> syntax for overriding this behaviour, i mean, obviously the complexity
> of supporting all operators with the getitem syntax could introduce
> alot of clutter. But maybe there's an elegant solution out there...
I don't think it needs a syntax for that, but I'm not so sure a method
to modify a value in place with a single key lookup wouldn't
occasioanally be useful.
For instance:
def increment(value):
return value+1
d = { 'a': 1 }
d.apply_to_value('a',increment)
print d
and d['a'] would be 2. The difference is that only one lookup
occurs. (The dictionary would be locked to modifications while the
callback function is called, same as if it were being iterated over.)
Thing is, there's no way to get that ability except subclass dict in
C; can't be done from Python. Which is why even though it'd be a
pretty rare need it at least deserves a bit of consideration before
being tabled.
As a workaround, if lookups are expensive, you can add an indirection
to the dict. For example, you could let each value in the dict be a
singleton list which is permanently bound to the key. Then you only
have look up the key once to modify the value.
d = { 'a': [1] }
ston = d['a']
ston[0] += 3
and now d['a'][0] is 4, and there was only one dict lookup. Downside
is that you have the [0] to carry around, but that is the price of
indirection. A wrapper class might help.
Carl Banks
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
In article <21e57363-4e92-41cb-9907-5aef96ad0...@o15g2000yqm.googlegroups.com>, RunThePun wrote: > >Anybody have any more ideas? I think python should/could havev a >syntax for overriding this behaviour, i mean, obviously the complexity >of supporting all operators with the getitem syntax could introduce >alot of clutter. But maybe there's an elegant solution out there... Any solution you create must support this: >>> L = [3] >>> L[0] += 7 >>> L [10] -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ "I support family values -- Addams family values" --www.nancybuttons.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On 2009-08-27 01:49 AM, RunThePun wrote: Anybody have any more ideas? I think python should/could havev a syntax for overriding this behaviour, i mean, obviously the complexity of supporting all operators with the getitem syntax could introduce alot of clutter. But maybe there's an elegant solution out there... I would recommend just adding a method to MyDict that does exactly what you want. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Aug 27, 6:58 am, Robert Kern wrote:
> On 2009-08-26 20:00 PM, Jan Kaliszewski wrote:
>
>
>
>
>
> > 27-08-2009 o 00:48:33 Robert Kern wrote:
>
> >> On 2009-08-26 17:16 PM, RunThePun wrote:
> >>> I'd like to build a database wrapper using DictMixin and allow items
> >>> to be appended by my own code. The problem is += is always understood
> >>> as setitem and getitem plainly.
>
> >>> d = MyDict()
> >>> d['a'] = 1
>
> >>> # this is the problem code that's I'd like to override. It's always
> >>> setitem('a', getitem('a') + 3)
> >>> d['a'] += 3
> >>> # i wanted to do something like my own 'appenditem' function which for
> >>> example could be useful if getitem is an expensive operation which can
> >>> be avoided.
>
> >>> I hope that was clear enough of a request, it's really late at night
> >>> here...
>
> >> I'm sorry, this is just part of the syntax of Python. You cannot
> >> override it.
>
> > Though
> > d['a'] = 3
> > is equivalent to:
> > d.__setitem__('a', 3)
>
> > The
> > d['a'] += 3
> > *is not* equivalent to:
> > d.__setitem__('a', d.__getitem__('a') + 3)
> > *but is* equivalent to:
> > d.__getitem__('a').__iadd__(3)
>
> > Then you can override __getitem__() of MyDict in such a way that it
> > returns prepared (wrapped) object with overriden __iadd__() as you
> > want to.
>
> You could, but then you will almost certainly run into problems using the
> wrapped object in places that really expect the true object.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless enigma
> that is made terrible by our own mad attempt to interpret it as though it
> had
> an underlying truth."
> -- Umberto Eco
Exactly my problem Robert. I'm actually going to be using this dict
for storing and retrieving strings so wrapping str and replacing the
iadd would cause alot of craziness.
Anybody have any more ideas? I think python should/could havev a
syntax for overriding this behaviour, i mean, obviously the complexity
of supporting all operators with the getitem syntax could introduce
alot of clutter. But maybe there's an elegant solution out there...
---RP
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On 2009-08-26 20:00 PM, Jan Kaliszewski wrote:
27-08-2009 o 00:48:33 Robert Kern wrote:
On 2009-08-26 17:16 PM, RunThePun wrote:
I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.
d = MyDict()
d['a'] = 1
# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.
I hope that was clear enough of a request, it's really late at night
here...
I'm sorry, this is just part of the syntax of Python. You cannot
override it.
Though
d['a'] = 3
is equivalent to:
d.__setitem__('a', 3)
The
d['a'] += 3
*is not* equivalent to:
d.__setitem__('a', d.__getitem__('a') + 3)
*but is* equivalent to:
d.__getitem__('a').__iadd__(3)
Then you can override __getitem__() of MyDict in such a way that it
returns prepared (wrapped) object with overriden __iadd__() as you
want to.
You could, but then you will almost certainly run into problems using the
wrapped object in places that really expect the true object.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
27-08-2009 o 00:48:33 Robert Kern wrote:
On 2009-08-26 17:16 PM, RunThePun wrote:
I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.
d = MyDict()
d['a'] = 1
# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.
I hope that was clear enough of a request, it's really late at night
here...
I'm sorry, this is just part of the syntax of Python. You cannot
override it.
Though
d['a'] = 3
is equivalent to:
d.__setitem__('a', 3)
The
d['a'] += 3
*is not* equivalent to:
d.__setitem__('a', d.__getitem__('a') + 3)
*but is* equivalent to:
d.__setitem__('a', d.__getitem__('a').__iadd__(3))
Then you can override __getitem__() of MyDict in such a way that it
returns prepared (wrapped) object with overriden __iadd__() as you
want to.
How could I now it:
1 import collections
2 import functools
3 import itertools
4
5
6 def verbose_func(func):
7 'Function decorator that makes a function "verbose"'
8
9 @functools.wraps(func, assigned=('__name__', '__doc__'))
10 def func_wrapper(*args, **kwargs):
11 iargs = (map(str, args))
12 ikwargs = ('{0}={1}'.format(key, value)
13for key, value in kwargs.items())
14 func_args = ', '.join(itertools.chain(iargs, ikwargs))
15 print('{0}({1})'.format(func.__name__, func_args))
16 return func(*args, **kwargs)
17
18 return func_wrapper
19
20
21 def verbose_cls(base):
22 'Class decorator that makes callable attributes "verbose"'
23
24 quiet = ('__new__', '__repr__', '__str__')
25
26 def cls_wrapper(cls):
27 for name in vars(base):
28 attr = getattr(cls, name)
29 if isinstance(attr, collections.Callable) and name not in
quiet:
30 setattr(cls, name, verbose_func(attr))
31 return cls
32
33 return cls_wrapper
34
35
36 @verbose_cls(dict)
37 class VerboseDict(dict):
38 pass
39
40
41 @verbose_cls(int)
42 class MyInt(int):
43
44 @verbose_func
45 def __iadd__(self, other):
46 int.__add__(self, other) # can do something more interesting
47
48
49 if __name__ == '__main__':
50 d = VerboseDict()
51
52 print("d['a'] = 3")
53 d['a'] = MyInt(3)
54
55 print("d['a'] += 3")
56 d['a'] += MyInt(3)
*j
--
Jan Kaliszewski (zuo)
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
27-08-2009 o 00:48:33 Robert Kern wrote:
On 2009-08-26 17:16 PM, RunThePun wrote:
I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.
d = MyDict()
d['a'] = 1
# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.
I hope that was clear enough of a request, it's really late at night
here...
I'm sorry, this is just part of the syntax of Python. You cannot
override it.
Though
d['a'] = 3
is equivalent to:
d.__setitem__('a', 3)
The
d['a'] += 3
*is not* equivalent to:
d.__setitem__('a', d.__getitem__('a') + 3)
*but is* equivalent to:
d.__getitem__('a').__iadd__(3)
Then you can override __getitem__() of MyDict in such a way that it
returns prepared (wrapped) object with overriden __iadd__() as you
want to.
How could I now it:
1 import collections
2 import functools
3 import itertools
4
5
6 def verbose_func(func):
7 'Function decorator that makes a function "verbose"'
8
9 @functools.wraps(func, assigned=('__name__', '__doc__'))
10 def func_wrapper(*args, **kwargs):
11 iargs = (map(str, args))
12 ikwargs = ('{0}={1}'.format(key, value)
13for key, value in kwargs.items())
14 func_args = ', '.join(itertools.chain(iargs, ikwargs))
15 print('{0}({1})'.format(func.__name__, func_args))
16 return func(*args, **kwargs)
17
18 return func_wrapper
19
20
21 def verbose_cls(base):
22 'Class decorator that makes callable attributes "verbose"'
23
24 quiet = ('__new__', '__repr__', '__str__')
25
26 def cls_wrapper(cls):
27 for name in vars(base):
28 attr = getattr(cls, name)
29 if isinstance(attr, collections.Callable) and name not in
quiet:
30 setattr(cls, name, verbose_func(attr))
31 return cls
32
33 return cls_wrapper
34
35
36 @verbose_cls(dict)
37 class VerboseDict(dict):
38 pass
39
40
41 @verbose_cls(int)
42 class MyInt(int):
43
44 @verbose_func
45 def __iadd__(self, other):
46 int.__add__(self, other) # can do something more interesting
47
48
49 if __name__ == '__main__':
50 d = VerboseDict()
51
52 print("d['a'] = 3")
53 d['a'] = MyInt(3)
54
55 print("d['a'] += 3")
56 d['a'] += MyInt(3)
*j
--
Jan Kaliszewski (zuo)
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
I haven't tested it, but did you encounter a problem defining __iadd__ in
the class definition?
See:
http://docs.python.org/reference/datamodel.html#object.__iadd__
Cheers,
Ching-Yun "Xavier" Ho, Technical Artist
Contact Information
Mobile: (+61) 04 3335 4748
Skype ID: SpaXe85
Email: cont...@xavierho.com
Website: http://xavierho.com/
On Thu, Aug 27, 2009 at 8:48 AM, Robert Kern wrote:
> On 2009-08-26 17:16 PM, RunThePun wrote:
>
>> I'd like to build a database wrapper using DictMixin and allow items
>> to be appended by my own code. The problem is += is always understood
>> as setitem and getitem plainly.
>>
>> d = MyDict()
>> d['a'] = 1
>>
>> # this is the problem code that's I'd like to override. It's always
>> setitem('a', getitem('a') + 3)
>> d['a'] += 3
>> # i wanted to do something like my own 'appenditem' function which for
>> example could be useful if getitem is an expensive operation which can
>> be avoided.
>>
>> I hope that was clear enough of a request, it's really late at night
>> here...
>>
>
> I'm sorry, this is just part of the syntax of Python. You cannot override
> it.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma
> that is made terrible by our own mad attempt to interpret it as though it
> had
> an underlying truth."
> -- Umberto Eco
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On 2009-08-26 17:16 PM, RunThePun wrote:
I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.
d = MyDict()
d['a'] = 1
# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.
I hope that was clear enough of a request, it's really late at night
here...
I'm sorry, this is just part of the syntax of Python. You cannot override it.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
--
http://mail.python.org/mailman/listinfo/python-list
Overriding iadd for dictionary like objects
I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.
d = MyDict()
d['a'] = 1
# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.
I hope that was clear enough of a request, it's really late at night
here...
thanks,
RunPun
--
http://mail.python.org/mailman/listinfo/python-list
