Re: Overriding iadd for dictionary like objects
On Sep 1, 3:00 am, a...@pythoncraft.com (Aahz) wrote:
In article
b11a8a0e-03ca-41c9-b0d0-c5180b6a2...@p15g2000vbl.googlegroups.com,
RunThePun ubershme...@gmail.com wrote:
On Aug 30, 10:33=A0pm, a...@pythoncraft.com (Aahz) wrote:
In article e09276e8-8152-4002-8366-4c12705a8...@l35g2000vba.googlegroups=
.com,
RunThePun =A0ubershme...@gmail.com wrote:
I made a DictMixin where the keys are filenames and the values are the
file contents. It was very simple and easy to do thanks to DictMixin.
For example this code writes abc in a file named temp.txt and
prints the contents of the file named swallow, these files are
looked up/created/deleted in the directory spam:
d =3D3D FilesDict('spam')
d['temp.txt'] =3D3D 'abc'
print(d['swallow'])
My problem arose when I wanted to append a string to a file which
using open(..., 'ab') would have been miles more efficient because I
wouldn't have to read the entire file (__getitem__) and then write the
entire file back (__setitem__). The files are expected to be as big as
600 KB which will be appended 30 bytes at a time about 3 times a
second. Performance-wise the system would probably work without open
(..., 'ab') but it would be a real thrashing so the current solution
uses a method AddTo as Robert suggested, sacrificing the neat
getitem/setitem syntax.
You can do mostly what you want, I think, by having __setitem__()
convert string values into FileProxy() objects that have an appropriate
__iadd__() method. =A0That brings a whole new set of problems, of course.
I'm guessing you meant __getitem__, which is what Jan Kaliszewski
suggested, but as you noted, would be a bit cumbersome in this case.
Actually, what I meant was __setitem__. The idea is that you create the
proxy item when you add the data to the dict (wrapping the original
data), and the proxy has an __iadd__ method, which would allow you to do
the file append.
--
Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/
I support family values -- Addams family values --www.nancybuttons.com
But you do mean that __getitem__ would return a wrapped object as
well, right? Otherwise I don't see how the iadd would be relevant
because:
d['a'] += 3
is equivalent to:
d.__setitem__('a', d.__getitem__('a').__iadd__(3))
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Aug 30, 10:33 pm, a...@pythoncraft.com (Aahz) wrote:
In article
e09276e8-8152-4002-8366-4c12705a8...@l35g2000vba.googlegroups.com,
RunThePun ubershme...@gmail.com wrote:
I made a DictMixin where the keys are filenames and the values are the
file contents. It was very simple and easy to do thanks to DictMixin.
For example this code writes abc in a file named temp.txt and
prints the contents of the file named swallow, these files are
looked up/created/deleted in the directory spam:
d =3D FilesDict('spam')
d['temp.txt'] =3D 'abc'
print(d['swallow'])
My problem arose when I wanted to append a string to a file which
using open(..., 'ab') would have been miles more efficient because I
wouldn't have to read the entire file (__getitem__) and then write the
entire file back (__setitem__). The files are expected to be as big as
600 KB which will be appended 30 bytes at a time about 3 times a
second. Performance-wise the system would probably work without open
(..., 'ab') but it would be a real thrashing so the current solution
uses a method AddTo as Robert suggested, sacrificing the neat
getitem/setitem syntax.
You can do mostly what you want, I think, by having __setitem__()
convert string values into FileProxy() objects that have an appropriate
__iadd__() method. That brings a whole new set of problems, of course.
--
Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/
I support family values -- Addams family values --www.nancybuttons.com
I'm guessing you meant __getitem__, which is what Jan Kaliszewski
suggested, but as you noted, would be a bit cumbersome in this case.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Aug 29, 1:58 pm, Carl Banks pavlovevide...@gmail.com wrote:
On Aug 28, 10:37 pm, Joshua Judson Rosen roz...@geekspace.com wrote:
Carl Banks pavlovevide...@gmail.com writes:
On Aug 28, 2:42 pm, Terry Reedy tjre...@udel.edu wrote:
Carl Banks wrote:
I don't think it needs a syntax for that, but I'm not so sure a method
to modify a value in place with a single key lookup wouldn't
occasioanally be useful.
Augmented assignment does that.
Internally uses two lookups, one for getting, and one for setting.
I think this is an unavoidable given Python's semantics. Look at
the traceback:
def x():
... d['a'] += 1
...
dis.dis(x)
2 0 LOAD_GLOBAL 0 (d)
3 LOAD_CONST 1 ('a')
6 DUP_TOPX 2
9 BINARY_SUBSCR
OK, there's one lookup, but...
10 LOAD_CONST 2 (1)
13 INPLACE_ADD
14 ROT_THREE
15 STORE_SUBSCR
16 LOAD_CONST 0 (None)
19 RETURN_VALUE
... I don't see anything in there that retrieves the value a second time
STORE_SUBSCR has to look up the position in the hash table to store
the value, hence the second lookup.
As a workaround, if lookups are expensive,
But they are not. Because (C)Python is heavily based on dict name lookup
for builtins and global names and attributes, as well as overt dict
lookup, must effort has gone into optimizing dict lookup.
The actual lookup algorithm Python dicts use is well-optimized, yes,
but the dict could contain keys that have expensive comparison and
hash-code calculation, in which case lookup is going to be slow.
I'll like the originator correct me if I've made a mistake, but I read
lookup as actually meaning lookup, not value-comparison.
This has nothing to do with value comparison. I was talking about key
comparison, which happens when looking up a position in a hash table.
I was the first person to use the word lookup in this thread and I
specifically meant hash-table position lookup.
At least in part because the question, as it was posed, specifically
related to a wrapper-class (providing a mapping (dict like) interface)
around a database of some sort other than Python's dict class per se.
How do the details of Python's native dict-type's internal (hashtable)
algorithm matter when they're explicitly /not/ being used?
Well it doesn't apply specifically to the OP's problem. I changed the
topic a bit by making it specific to dicts. Is that ok with you? Was
that not allowed?
The OP can add a method like apply_to_value to his own class, but one
can't do that for dicts. Ergo why something like apply_to_value()
would be useful enough in rare circumstances where lookup is very slow
to merit a moments consideration before being rejected.
(If dict did have a method like that, the OP would at least know which
method to override.)
Carl Banks
First of all I'd like to say thanks for this discussion, you guys are
awesome.
I probably should have explained my problem better to begin with and I
apologize for that. So now I'll start from the top:
I made a DictMixin where the keys are filenames and the values are the
file contents. It was very simple and easy to do thanks to DictMixin.
For example this code writes abc in a file named temp.txt and
prints the contents of the file named swallow, these files are
looked up/created/deleted in the directory spam:
d = FilesDict('spam')
d['temp.txt'] = 'abc'
print(d['swallow'])
This was very convenient for me because I wanted to use a simple DB
which could be read and edited by shell scripts and non-pythonistas,
without the heavy ORM. Also, if in the future an online full featured
DB would be needed, I could easily convert the DictMixin methods. So
up to here I had a good solution.
My problem arose when I wanted to append a string to a file which
using open(..., 'ab') would have been miles more efficient because I
wouldn't have to read the entire file (__getitem__) and then write the
entire file back (__setitem__). The files are expected to be as big as
600 KB which will be appended 30 bytes at a time about 3 times a
second. Performance-wise the system would probably work without open
(..., 'ab') but it would be a real thrashing so the current solution
uses a method AddTo as Robert suggested, sacrificing the neat
getitem/setitem syntax.
Just so I don't leave out any information, I actually also made a
DirectoryDict for having multiple 'tables'. In this DictMixin, keys
are directory names and values are FilesDict instances. So to write
European or African to the file /root/tmp/velocity one would be
just:
d = DirectoryDict(/root)
d[tmp][velocity] = European or African
So now I hope it's clearer why and how I wanted a special
__item_iadd__ for the dictionary syntax,
RunThePun
Re: Overriding iadd for dictionary like objects
On Aug 27, 6:58 am, Robert Kern robert.k...@gmail.com wrote:
On 2009-08-26 20:00 PM, Jan Kaliszewski wrote:
27-08-2009 o 00:48:33 Robert Kern robert.k...@gmail.com wrote:
On 2009-08-26 17:16 PM, RunThePun wrote:
I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.
d = MyDict()
d['a'] = 1
# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.
I hope that was clear enough of a request, it's really late at night
here...
I'm sorry, this is just part of the syntax of Python. You cannot
override it.
Though
d['a'] = 3
is equivalent to:
d.__setitem__('a', 3)
The
d['a'] += 3
*is not* equivalent to:
d.__setitem__('a', d.__getitem__('a') + 3)
*but is* equivalent to:
d.__getitem__('a').__iadd__(3)
Then you can override __getitem__() of MyDict in such a way that it
returns prepared (wrapped) object with overriden __iadd__() as you
want to.
You could, but then you will almost certainly run into problems using the
wrapped object in places that really expect the true object.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it
had
an underlying truth.
-- Umberto Eco
Exactly my problem Robert. I'm actually going to be using this dict
for storing and retrieving strings so wrapping str and replacing the
iadd would cause alot of craziness.
Anybody have any more ideas? I think python should/could havev a
syntax for overriding this behaviour, i mean, obviously the complexity
of supporting all operators with the getitem syntax could introduce
alot of clutter. But maybe there's an elegant solution out there...
---RP
--
http://mail.python.org/mailman/listinfo/python-list
Overriding iadd for dictionary like objects
I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.
d = MyDict()
d['a'] = 1
# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.
I hope that was clear enough of a request, it's really late at night
here...
thanks,
RunPun
--
http://mail.python.org/mailman/listinfo/python-list
