Re: [ZODB-Dev] Indexing and dates/times

2010-07-13 Thread Jim Fulton
On Tue, Jul 13, 2010 at 4:35 AM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: Hello, I am currently trying to devise a way to index and retrieve some millions of objects according to their modification date/time. One of the problems I'm facing is that of index granularity: I'd like to

Re: [ZODB-Dev] Indexing and dates/times

2010-07-13 Thread Benji York
On Tue, Jul 13, 2010 at 4:35 AM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: Hello, I am currently trying to devise a way to index and retrieve some millions of objects according to their modification date/time. So, in a relational DB i would do something like: SELECT * FROM table

Re: [ZODB-Dev] Indexing and dates/times

2010-07-13 Thread Pedro Ferreira
So, the issue is that you have multiple items with the same key. This is simply handled by using sets as values ion a BTree. There are existing index implementations that do this. Hmm... no, in fact the problem is that most of the time I will have only one value per index entry.So, in a

Re: [ZODB-Dev] Indexing and dates/times

2010-07-13 Thread Pedro Ferreira
If you use the timestamp as the key and you want to retrieve all values between two timestamps (inclusive), you can do my_btree.values(min=start, max=end) Yes, but, as I mentioned in my answer to Jim's mail, my concern is the performance of this range operation for a very large

Re: [ZODB-Dev] Indexing and dates/times

2010-07-13 Thread Jim Fulton
On Tue, Jul 13, 2010 at 7:51 AM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: ... Hmm... no, in fact the problem is that most of the time I will have only one value per index entry.So, in a relational DB i would do something like: As I mentioned in some text you didn't quote, you could use

[ZODB-Dev] Indexing and dates/times

2010-07-12 Thread Pedro Ferreira
Hello all, I am currently trying to devise a way to index and retrieve some millions of objects according to their modification date/time. One of the problems I'm facing is that of index granularity: I'd like to provide to the second granularity, but for that I need some structure that lets

Re: [ZODB-Dev] Indexing and dates/times

2010-07-12 Thread Jim Fulton
On Mon, Jul 12, 2010 at 11:47 AM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: Hello all, I am currently trying to devise a way to index and retrieve some millions of objects according to their modification date/time. One of the problems I'm facing is that of index granularity: I'd like