Re: [ZODB-Dev] grab zodb dict keys by search term
And that is exactly what I was looking for! Thanks! Tamer On 11.12.2013 14:27, Vincent Pelletier wrote: On Wed, 11 Dec 2013 12:27:19 +0200, Tamer Higazi tamerito...@arcor.de wrote : If i would have 100.000 - 500.000 entries, where 150 would match, I don't want to go over those completely. So, if there is a pythonic way to get it solved on a performant way, or a ZODB way as well that would be wondefull. BTrees have a very nice API, which allow scanning only a subset of keys by providing min and max boundaries: for key, value in some_btree.items( min='aa', max='b', excludemax=True): # Do stuff should do what you want. http://pythonhosted.org/BTrees/ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] grab zodb dict keys by search term
On Wed, Dec 11, 2013 at 3:27 AM, Tamer Higazi tamerito...@arcor.de wrote: Perhaps it's my fault, the way I stored the data. The point is, that I am looking the fastest and performant way to grab the data from a big pool. I do not think folks are suggesting you iterate over the big pool keys/values/items. Rather, folks are suggesting that you consider using BTrees to make your own indexes, if you choose not to use search/catalog/indexing tools like Hypatia or repoze.catalog (with or without Souper). For each field of data you care to search, you need an index that can yield identifiers that match the keys in your big pool of data (which I assume is also stored in a BTree). I personally chose to solve this problem by building myself a small container/retrieval framework uses a combination of BTrees, UUID usage, zope.schema, and repoze.catalog to tackle problems (this assumes that all my items are keyed by UUIDs). Example: https://github.com/upiq/uu.retrieval I don't want to iterate the whole thing over and over again to get the matched entries. If i would have 100.000 - 500.000 entries, where 150 would match, I don't want to go over those completely. You need indexes, either use a built-in ZODB-native catalog/search framework like repoze.catalog/hypatia (or Souper), external search service like Solr/ElasticSearch, or build your own indexes using two BTrees each (forward mapping of values to lists of matching ids, reverse mapping of ids to values). So, if there is a pythonic way to get it solved on a performant way, or a ZODB way as well that would be wondefull. You need indexes; most folks choose to use some sort of library to solve this problem instead of DIY indexes, but that's up to you. Sean ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] grab zodb dict keys by search term
On Wed, 11 Dec 2013 12:27:19 +0200, Tamer Higazi tamerito...@arcor.de wrote : If i would have 100.000 - 500.000 entries, where 150 would match, I don't want to go over those completely. So, if there is a pythonic way to get it solved on a performant way, or a ZODB way as well that would be wondefull. BTrees have a very nice API, which allow scanning only a subset of keys by providing min and max boundaries: for key, value in some_btree.items( min='aa', max='b', excludemax=True): # Do stuff should do what you want. http://pythonhosted.org/BTrees/ -- Vincent Pelletier ERP5 - open source ERP/CRM for flexible enterprises ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] grab zodb dict keys by search term
you might be interested in https://pypi.python.org/pypi/souper/ which addresses this kind of problem hth Jens On 2013-12-10 23:46, Tamer Higazi wrote: Hi people! I am working a lot with IOB and OOB Trees as well with PersistentDict and PersistentList to store my Data in ZODB. I want to ask if there is a way to grab data from a dataset by search time. I know only the SQL way: select lower(COL1) from table1 where COL1 LIKE 'aa%' so usually I want to get the entryies only that starts with 'aa' do I have to iterate the entire dict with an interator or are there builtin functions that could accomplish those tasks for me ?! I would kindly thank you Tamer ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev -- Klein Partner KG, member of BlueDynamics Alliance ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] grab zodb dict keys by search term
ZODB has nothing built-in. You build it on top of the ZODB. I think what you are looking for is something like Hypatia - https://github.com/Pylons/hypatia On Tue, Dec 10, 2013 at 4:46 PM, Tamer Higazi tamerito...@arcor.de wrote: Hi people! I am working a lot with IOB and OOB Trees as well with PersistentDict and PersistentList to store my Data in ZODB. I want to ask if there is a way to grab data from a dataset by search time. I know only the SQL way: select lower(COL1) from table1 where COL1 LIKE 'aa%' so usually I want to get the entryies only that starts with 'aa' do I have to iterate the entire dict with an interator or are there builtin functions that could accomplish those tasks for me ?! I would kindly thank you Tamer ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Skype/Twitter:: runyaga Office:: 713.942.2377 ext 111 http://ploud.com/ Plone site in less than 10 seconds ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] grab zodb dict keys by search term
Hi Tamer,
It depends on what you're trying to achieve. BTrees should not be mutated
whilst iterating over them, so depending on how you have structured your
data, you might do something like:
for key in [x for x in BTree.keys() if x.startswith('aa')]:
# Do stuff
Or perhaps something else like:
for key in list(BTree.keys()):
if key['subkey'] == 5:
yield BTree[key]
...Or some other pattern depending on your use case.
The key here is that the ZODB really allows you to cut out the middle
man where previously you might store data in an RDBMS and then do
processing/transformation of that data in Python after extracting it with
SQL (and then re-storing it again afterwards), instead now you can just
directly process the data as if it was in Python objects waiting the
entire time.
Hope this helps.
regards,
Darryl Dixon
Winterhouse Consulting Ltd
Hi people!
I am working a lot with IOB and OOB Trees as well with PersistentDict
and PersistentList to store my Data in ZODB.
I want to ask if there is a way to grab data from a dataset by search
time.
I know only the SQL way:
select lower(COL1) from table1 where COL1 LIKE 'aa%'
so usually I want to get the entryies only that starts with 'aa'
do I have to iterate the entire dict with an interator or are there
builtin functions that could accomplish those tasks for me ?!
I would kindly thank you
Tamer
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
