Ahmed Barakat wrote:
In case I have a huge datastore (10000 entries, each entry has like 6
properties), what is the best way
to handle the search within such a huge datastore, and what if I want to
make a generic search, for example
you write a word and i use it to search within all properties I have for all
entries?
Is the conversion to XML a good solution, or it is not?
sorry for being new to web development, and python.
Thanks in advance.
I don't see anything about your query which is specific to web
development, and there's no need to be apologetic for being new anyway.
One person's "huge" is another person's "pretty large." I'd say 10000
items is pretty small if you're working on the desktop, as you can
readily hold all the data in "memory." I edit text files bigger than
that. But I'll assume your data really is huge, or will grow to be
huge, or is an environment which treats it as huge.
When you're parsing large amounts of data, there are always tradeoffs
between performance and other characteristics, usually size and
complexity. If you have lots of data, you're probably best off by using
a standard code system -- a real database. The developers of such
things have decades of experience in making certain things fast,
reliable, and self-consistent.
But considering only speed here, I have to point out that you have to
understand databases, and your particular model of database, pretty well
to really benefit from all the performance tricks in there. Keeping it
abstract, you specify what parts of the data you care about fast random
access to. If you want fast search access to "all" of it, your database
will generally be huge, and very slow to updates. And the best way to
avoid that is to pick a database mechanism that best fits your search
mechanism. I hate to think how many man-centuries Google has dedicated
to getting fast random word access to its *enormous* database. I'm sure
they did not build on a standard relational model.
If you plan to do it yourself, I'd say the last thing you want to do is
use XML. XML may be convenient way to store self-describing data, but
it's not quick to parse large amounts of it. Instead, store the raw
data in text form, with separate index files describing what is where.
Anything that's indexed will be found rapidly, while anything that isn't
will require search of the raw data.
There are algorithms for searching raw data that are faster than
scanning every byte, but a relevant index will almost always be faster.
DaveA
--
http://mail.python.org/mailman/listinfo/python-list
