Re: Stored fields: decompression slows down in my scenario ... any idea for a workaround?

2013-06-25 Thread Adrien Grand
Hi, On Mon, Jun 24, 2013 at 6:13 PM, Mathias Lux wrote: > When searching for an image within memory I came down to 44ms. > Therefore, 77ms is totally acceptable in these terms. My benchmarking > of the BinaryDocValuesField showed that it'd come close to the 44ms, > but I didn't go for a full eval

Re: Stored fields: decompression slows down in my scenario ... any idea for a workaround?

2013-06-24 Thread Mathias Lux
Hi! Thanks again for all the help. Seems like the field compression allows a huge step forward for my case. Here's some benchmarking for those of you interested: a document is * a StringField giving the actual image path * a single 64 byte feature (global OpponentHistogram) number of documents

Re: Stored fields: decompression slows down in my scenario ... any idea for a workaround?

2013-06-24 Thread Adrien Grand
Hi, On Mon, Jun 24, 2013 at 2:47 PM, Mathias Lux wrote: > Still, I've read that all the BinaryDocValues go directly to memory. > Am I right with this? It is true that the current default implementation stores them in memory. However, disk doc values formats can be configured on a per-field basis

Re: Stored fields: decompression slows down in my scenario ... any idea for a workaround?

2013-06-24 Thread Mathias Lux
Hi! I'm basically in the midth of experiments. The idea with the BinaryDocValuesField worked great, it's blazing fast ;) Reading 49,904 documents, each with a 64 byte value got down to 33 ms from 774 (with StoredField). Also writing is much faster 9ms vs. 22ms. Still, I've read that all the Binar

Re: Stored fields: decompression slows down in my scenario ... any idea for a workaround?

2013-06-24 Thread Adrien Grand
Hi, On Sun, Jun 23, 2013 at 9:08 PM, Savia Beson wrote: > I think Mathias was talking about the case with many smallish fields that all > get read per document. DV approach would mean seeking N times, while stored > fields, only once? Or you meant he should encode all his fields into single

Re: Stored fields: decompression slows down in my scenario ... any idea for a workaround?

2013-06-24 Thread Mathias Lux
> Uwe >> >> - >> Uwe Schindler >> H.-H.-Meier-Allee 63, D-28213 Bremen >> http://www.thetaphi.de >> eMail: u...@thetaphi.de >> >> >>> -----Original Message- >>> From: mathias@gmail.com [mailto:mathias@gmail.com] On

Re: Stored fields: decompression slows down in my scenario ... any idea for a workaround?

2013-06-23 Thread Savia Beson
aphi.de > eMail: u...@thetaphi.de > > >> -Original Message- >> From: mathias@gmail.com [mailto:mathias@gmail.com] On Behalf Of >> Mathias Lux >> Sent: Sunday, June 23, 2013 7:27 PM >> To: java-user@lucene.apache.org >> Subject: Stored fields:

RE: Stored fields: decompression slows down in my scenario ... any idea for a workaround?

2013-06-23 Thread Uwe Schindler
il.com [mailto:mathias@gmail.com] On Behalf Of > Mathias Lux > Sent: Sunday, June 23, 2013 7:27 PM > To: java-user@lucene.apache.org > Subject: Stored fields: decompression slows down in my scenario ... any idea > for a workaround? > > Hi! > > I'm man

Stored fields: decompression slows down in my scenario ... any idea for a workaround?

2013-06-23 Thread Mathias Lux
Hi! I'm managing the development of LIRE (https://code.google.com/p/lire/), a image search toolbox based on Lucene. While optimizing different search routines for global image features I came around to take a look at the CPU usage, i.e. to see if my new distance function is faster than the old one