thanks. I'd like trying it and do some experiment on our dataset. 2010/12/15 Michael McCandless <luc...@mikemccandless.com>: > Hi Li Li, > > That issue has such a big patch, and enough of us are now iterating on > it, that we cut a dedicated branch for it. > > But note that this branch is off of trunk (to be 4.0). > > You should be able to do this: > > svn checkout > https://svn.apache.org/repos/asf/lucene/dev/branches/bulkpostings > > And then run things in there. I just committed FOR/PFOR prototype > codecs from LUCENE-1410 onto that branch, so eg you can run unit tests > using those codecs by running "ant test > -Dtests.codec=PatchedFrameOfRef". > > Please post patches back if you improve things! We need all the help > we can get :) > > Mike > > On Wed, Dec 15, 2010 at 5:54 AM, Li Li <fancye...@gmail.com> wrote: >> hi Michael >> you posted a patch here https://issues.apache.org/jira/browse/LUCENE-2723 >> I am not familiar with patch. do I need download >> LUCENE-2723.patch(there are many patches after this name, do I need >> the latest one?) and LUCENE-2723_termscorer.patch and patch them >> (patch -p1 <LUCENE-2723.patch)? I just check out the latest source >> code from http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene >> >> >> 2010/12/14 Michael McCandless <luc...@mikemccandless.com>: >>> Likely you are seeing the startup cost of hotspot compiling the PFOR code? >>> >>> Ie, does your test first "warmup" the JRE and then do the real test? >>> >>> I've also found that running -Xbatch produces more consistent results >>> from run to run, however, those results may not be as fast as running >>> w/o -Xbatch. >>> >>> Also, it's better to test on actual data (ie a Lucene index's >>> postings), and in the full context of searching, because then we get a >>> sense of what speedups a real app will see... micro-benching is nearly >>> impossible in Java since Hotspot acts very differently vs the "real" >>> test. >>> >>> Mike >>> >>> On Tue, Dec 14, 2010 at 2:50 AM, Li Li <fancye...@gmail.com> wrote: >>>> Hi >>>> I tried to integrate PForDelta into lucene 2.9 but confronted a problem. >>>> I use the implementation in >>>> http://code.google.com/p/integer-array-compress-kit/ >>>> it implements a basic PForDelta algorithm and an improved one(which >>>> called NewPForDelta, but there are many bugs and I have fixed them), >>>> But compare it with VInt and S9, it's speed is very slow when only >>>> decode small number of integer arrays. >>>> e.g. when I decoded int[256] arrays which values are randomly >>>> generated between 0 and 100, if decode just one array. PFor(or >>>> NewPFor) is very slow. when it continuously decodes many arrays such >>>> as 10000, it's faster than s9 and vint. >>>> Another strange phenomena is that when call PFor decoder twice, the >>>> 2nd times it's faster. Or I call PFor first then NewPFor, the NewPFor >>>> is faster. reverse the call sequcence, the 2nd called decoder is >>>> faster >>>> e.g. >>>> ct.testNewPFDCodes(list); >>>> ct.testPFor(list); >>>> ct.testVInt(list); >>>> ct.testS9(list); >>>> >>>> NewPFD decode: 3614705 >>>> PForDelta decode: 17320 >>>> VINT decode: 16483 >>>> S9 decode: 19835 >>>> when I call by the following sequence >>>> >>>> ct.testPFor(list); >>>> ct.testNewPFDCodes(list); >>>> ct.testVInt(list); >>>> ct.testS9(list); >>>> >>>> PForDelta decode: 3212140 >>>> NewPFD decode: 19556 >>>> VINT decode: 16762 >>>> S9 decode: 16483 >>>> >>>> My implementation is -- group docIDs and termDocFreqs into block >>>> which contains 128 integers. when SegmentTermDocs's next method >>>> called(or read readNoTf).it decodes a block and save it to a cache. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: dev-h...@lucene.apache.org >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
