Re: anyone has interests about mg4j's new integer compression algorithm?

2012-07-06 Thread Dawid Weiss
I've repeated Sebastiano's experiments (and so did he). A few quotes from the communication. The index appears to be larger now--43.1GB. Probably they have better skipping structures that take more space. From what I can see the format is the same as before--the .frq file contains document

Re: anyone has interests about mg4j's new integer compression algorithm?

2012-07-06 Thread Li Li
I can understand these quotes. what's the conclusion from your communication? On Fri, Jul 6, 2012 at 4:20 PM, Dawid Weiss dawid.we...@cs.put.poznan.pl wrote: I've repeated Sebastiano's experiments (and so did he). A few quotes from the communication. The index appears to be larger

Re: anyone has interests about mg4j's new integer compression algorithm?

2012-07-06 Thread Dawid Weiss
That 4.0 is significantly faster than 3.6 for this benchmark and there were minor glitches in the benchmarking code itself. Dawid On Fri, Jul 6, 2012 at 11:47 AM, Li Li fancye...@gmail.com wrote: I can understand these quotes. what's the conclusion from your communication? On Fri, Jul 6, 2012

Re: anyone has interests about mg4j's new integer compression algorithm?

2012-07-06 Thread Robert Muir
I reviewed the benchmarking code on his website very quickly: * I don't like his NullCollector, it sets acceptsDocsOutOfOrder() = false, but its doing nothing but counting. By returning false here, he is declaring that the collector cares about docid order (which it doesnt), and preventing the

Re: anyone has interests about mg4j's new integer compression algorithm?

2012-06-24 Thread Dawid Weiss
Fyi. I contacted Sebastiano and will get hold of the data set and benchmarks he used to repeat his experiment with current trunk (curiosity). Any hints on which configuration should be used will be welcome. Dawid On Sat, Jun 23, 2012 at 12:38 PM, Li Li fancye...@gmail.com wrote: