Dennis Lee Bieber wrote:
On Thu, 29 Apr 2010 11:38:28 +0200, Karin Lagesen
karin.lage...@bio.uio.no declaimed the following in comp.lang.python:
Hello.
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is
Karin Lagesen wrote:
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is present
within the 83 million strings.
[...]
I run out of memory building both the set and the dictionary, so
what I seem to be left with is
In article 877hnpjtdw@rudin.co.uk,
Paul Rudin paul.nos...@rudin.co.uk wrote:
Karin Lagesen karin.lage...@bio.uio.no writes:
Hello.
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is present
within the 83
http://www.swizwatch.com/
All Cartier replica watches sold at Hotwristwatch.com are brand-new and high
quality. Each Cartier Replica Watch produced is examined carefully by our
quality test department and each watch is inspected again before being sent
to our customer. It is our desire that you do
Dennis Lee Bieber wrote:
On Thu, 29 Apr 2010 11:38:28 +0200, Karin Lagesen
karin.lage...@bio.uio.no declaimed the following in comp.lang.python:
Hello.
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is
Duncan Booth, 30.04.2010 10:20:
So more than 3GB just for the strings (and that's for Python 2.x on
Python 3.x you'll need nearly 5GB).
Running on a 64 bit version of Python should be fine, but for a 32 bit
system a naive approach just isn't going to work.
Option 1: use a trie. That should
Dennis Lee Bieber wrote:
On Sat, 01 May 2010 13:48:02 +0200, News123 news1...@free.fr declaimed
the following in gmane.comp.python.general:
Dennis Lee Bieber wrote:
That lets you do a binary search on the file. Much faster than a
linear search (linear search will average out to 41.5M
Karin Lagesen karin.lage...@bio.uio.no writes:
Hello.
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is present
within the 83 million strings.
Now, I have tried storing these strings as a list, a set and a
Paul Rudin paul.nos...@rudin.co.uk wrote:
Shouldn't a set with 83 million 14 character strings be fine in memory
on a stock PC these days? I suppose if it's low on ram you might start
swapping which will kill performance. Perhaps the method you're using
to build the data structures creates
On Fri, 30 Apr 2010 08:23:39 +0100, Paul Rudin wrote:
Karin Lagesen karin.lage...@bio.uio.no writes:
Hello.
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is present
within the 83 million strings.
Now, I
Duncan Booth duncan.bo...@invalid.invalid writes:
Paul Rudin paul.nos...@rudin.co.uk wrote:
Shouldn't a set with 83 million 14 character strings be fine in memory
on a stock PC these days? I suppose if it's low on ram you might start
swapping which will kill performance. Perhaps the method
s = 12345678901234
assert len(s) == 14
import sys
sys.getsizeof(s)
38
So a single 14 char string takes 38 bytes.
Make that at least 40 bytes. You have to take memory alignment into account.
So a set with 83000 such strings takes approximately 1 MB. So far fairly
trivial. But that's just the
I think one could apply an external hashing technique which would require only
very few disk accesses per lookup.
Unfortunately, I'm now aware of an implementation in Python.
Does anybody know about a Python implementation of external hashing?
Thanks,
Helmut.
--
Helmut Jarausch
Lehrstuhl fuer
On 04/30/2010 12:51 PM, Helmut Jarausch wrote:
I think one could apply an external hashing technique which would require only
very few disk accesses per lookup.
Unfortunately, I'm now aware of an implementation in Python.
Does anybody know about a Python implementation of external hashing?
Helmut Jarausch wrote:
I think one could apply an external hashing technique which would require only
very few disk accesses per lookup.
Unfortunately, I'm now aware of an implementation in Python.
Does anybody know about a Python implementation of external hashing?
Thanks,
Helmut.
That's
Hello.
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is present
within the 83 million strings.
Now, I have tried storing these strings as a list, a set and a dictionary.
I know that finding things in a set and a
Karin Lagesen wrote:
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is present
within the 83 million strings.
Now, I have tried storing these strings as a list, a set and a dictionary.
I know that finding
Karin Lagesen, 29.04.2010 11:38:
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is present
within the 83 million strings.
Now, I have tried storing these strings as a list, a set and a dictionary.
I know that
Karin Lagesen karin.lage...@bio.uio.no wrote in message
news:416f727c6f5b0edb932b425db9579808.squir...@webmail.uio.no...
Hello.
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is present
within the 83 million
Karin Lagesen wrote:
Hello.
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is present
within the 83 million strings.
Now, I have tried storing these strings as a list, a set and a dictionary.
I know that finding
MRAB pyt...@mrabarnett.plus.com wrote:
Karin Lagesen wrote:
Hello.
I have approx 83 million strings, all 14 characters long. I need to
be able to take another string and find out whether this one is
present within the 83 million strings.
Now, I have tried storing these strings as a
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is present
within the 83 million strings.
Have a look at the shelve module.
If you want to write the algorithm yourself, I suggest
http://en.wikipedia.org/wiki/Trie
On 4/29/2010 5:38 AM, Karin Lagesen wrote:
Hello.
I have approx 83 million strings, all 14 characters long. I need to be
able to take another string and find out whether this one is present
within the 83 million strings.
If the 'other string' is also 14 chars, so that you are looking for
23 matches
Mail list logo
