Raymond Hettinger wrote:
It might be better to give more generic advice that tends to be true
across implementations and versions: Dense collections like lists
tuples iterate faster than sparse structures like dicts and sets.
Whenever repeated iteration starts to dominate application
Hello,
I thought about another reason to resize the hash table when it has
too few elements. It's not only a matter of memory usage, it's also a
matter of time usage: iteration over a set or a dict requires going
over all the table. For example, iteration over a set which once had
1,000,000
[Noam]
For example, iteration over a set which once had
1,000,000 members and now has 2 can take 1,000,000 operations every
time you traverse all the (2) elements.
Do you find that to be a common or plausible use case?
Was Guido's suggestion of s=set(s) unworkable for some reason? dicts
and
On 12/31/05, Raymond Hettinger [EMAIL PROTECTED] wrote:
[Noam]
For example, iteration over a set which once had
1,000,000 members and now has 2 can take 1,000,000 operations every
time you traverse all the (2) elements.
Do you find that to be a common or plausible use case?
I don't have
Raymond Hettinger wrote:
Was Guido's suggestion of s=set(s) unworkable for some reason? dicts
and sets emphasize fast lookups over fast iteration -- apps requiring
many iterations over a collection may be better off converting to a list
(which has no dummy entries or empty gaps between
Noam Raphael [EMAIL PROTECTED] wrote:
On 12/31/05, Raymond Hettinger [EMAIL PROTECTED] wrote:
[Noam]
For example, iteration over a set which once had
1,000,000 members and now has 2 can take 1,000,000 operations every
time you traverse all the (2) elements.
Do you find that to
[Noam Raphael]
For example, iteration over a set which once had
1,000,000 members and now has 2 can take 1,000,000 operations every
time you traverse all the (2) elements.
[Raymond Hettinger]
Do you find that to be a common or plausible use case?
[Naom]
I don't have a concrete example in
[Noam]
For example, iteration over a set which once had
1,000,000 members and now has 2 can take 1,000,000 operations
every
time you traverse all the (2) elements.
Do you find that to be a common or plausible use case?
I don't have a concrete example in this minute, but a
[Fernando Perez]
Note that this is not a comment on the current discussion per se, but
rather a
small request/idea in the docs department: I think it would be a
really
useful
thing to have a summary page/table indicating the complexities of the
various
operations on all the builtin types,
Martin v. Löwis wrote:
Adal Chiriliuc wrote:
MSVC 7.1 and 8.0 malloc always uses the Windows heap functions
(HeapAlloc friends) if running on Windows 2000 or newer
(malloc.c and heapinit.c).
So it seems that for both Linux (gcc) and Win (msvc) the memory is
released to the operating
On 12/29/05, Noam Raphael [EMAIL PROTECTED] wrote:
On 12/29/05, Donovan Baarda [EMAIL PROTECTED] wrote:
Without some sort of fancy overkill size hinting or history tracking,
that's probably as good a heuristic as you can get.
I'm sorry, but it's not correct. There's a simple resize
Noam Raphael wrote:
I'm not saying that practically it must be used - I'm just saying that
it can't be called a heuristic, and that it doesn't involve any fancy
overkill size hinting or history tracking. It actually means
something like this:
1. If you want to insert and the table is full,
On Thu, 2005-12-29 at 17:17 +0100, Fredrik Lundh wrote:
Noam Raphael wrote:
I'm not saying that practically it must be used - I'm just saying that
it can't be called a heuristic, and that it doesn't involve any fancy
overkill size hinting or history tracking. It actually means
something
On 12/29/05, Fredrik Lundh [EMAIL PROTECTED] wrote:
Noam Raphael wrote:
I'm not saying that practically it must be used - I'm just saying that
it can't be called a heuristic, and that it doesn't involve any fancy
overkill size hinting or history tracking. It actually means
something like
Hello,
If I do something like this:
s = set()
for i in xrange(100):
s.add(i)
while s:
s.pop()
gc.collect()
the memory consumption of the process remains the same even after the pops.
I checked the code (that's where I started from, really), and there's
nothing in set.pop or
Noam Raphael wrote:
Should something be done about it?
It's very difficult to do something useful about it. Even if
you manage to determine how much memory you want to release,
it's nearly impossible to actually release the memory to the
operating system, because of the many layers of memory
On 12/29/05, Martin v. Löwis [EMAIL PROTECTED] wrote:
Noam Raphael wrote:
Should something be done about it?
It's very difficult to do something useful about it. Even if
you manage to determine how much memory you want to release,
it's nearly impossible to actually release the memory to the
It's very difficult to do something useful about it. Even if
you manage to determine how much memory you want to release,
it's nearly impossible to actually release the memory to the
operating system, because of the many layers of memory
management software that all need to agree that
Noam Raphael wrote:
I checked - when doing the same thing with lists, all the memory was
released for use by other Python objects, and most of it was released
for use by the operating system.
In this specific case, perhaps. malloc will typically return memory to
the system only if that memory
On 12/29/05, Raymond Hettinger [EMAIL PROTECTED] wrote:
What could be done is to add a test for excess dummy entries and trigger
a periodic resize operation. That would make the memory available for
other parts of the currently running script and possibly available for
the O/S.
The downside
Noam Raphael wrote:
The computer scientist in me prefers O() terms over changes in a
constant factor, but that's only me.
That remark, I don't understand. In a hash table, most simple
operations are O(n) as the worst-case time, except for operations
that may cause resizing, which are O(n**2)
On Thursday, December 29, 2005 Martin v. Löwis wrote:
Noam Raphael wrote:
In this specific case, perhaps. malloc will typically return memory to
the system only if that memory is at the end of the heap. If there
is more memory after block to be released, it can't return the memory
block,
I did a little test using MSVC 8.0 on WinXP.
I allocated 128 MB using 128 different buffers of 1 MB each,
freed half of them (alternatively) then freed the remaining half.
I monitored memory usage using the Task Manager and memory is really
freed as indicated by both the Mem Usage and VM Size
Noam Raphael [EMAIL PROTECTED] wrote:
On 12/29/05, Raymond Hettinger [EMAIL PROTECTED] wrote:
What could be done is to add a test for excess dummy entries and trigger
a periodic resize operation. That would make the memory available for
other parts of the currently running script and
24 matches
Mail list logo
