[ https://issues.apache.org/jira/browse/LUCENE-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636730#action_12636730 ]
Paul Elschot commented on LUCENE-1410: -------------------------------------- The number of bits is not really informative, it would be better to have a distribution of the result of getNumFrameBits(). Then we can see for which nrs of bits loop unrolling might be tried. As for decompression speed, please remember that the patching code (that decodes the exceptions into the output) has not yet been optimized at all. The lucene .frq file contains the docids as deltas and the frequencies but with a special encoding in case the frequency is one. I'd rather try compressing the real delta docids and the real frequencies than the encoded versions. The .prx file should be useable as it is, and from the results reported in the articles I would expect a real performance advantage for PFor for that. Michael, could you post a distribution of the number of frame bits for the .prx file? I'd like to know a reasonable maximum for unrolling the corresponding loops. >: maybe I'm doing something wrong I don't think so, the code is still young. Try running TestPFor and have a look at the output near the end for the case of 6 frame bits. That should show the unrolled decoding speed for the case without exceptions. Then compare that to the cases with lower and higher nrs of frame bits. >: Stepping back a bit: I wonder what %tg of the time in a "typical" Lucene search is spent decoding vInts? That would bound how much improvement we could ever expect from this optimization during searching. A: there is also the influence of the reduction of data to be fetched (via various caches) from the index. As reported in the articles, PFor strikes a nice optimum between decompression speed and fetching speed from (fast) disks. >: I was thinking local CPU's native asm. A: I'd try a C version first. Iirc gcc has a decent optimizer for bit ops, but it's been a while for me that I used C. For the record, to show the variation in decompression speeds for different numbers of frame bits without exceptions, here is my current output from TestPFor: {noformat} test901PerfFor1Decompress starting Compressed 128 bytes into 8, ratio 0.0625 test901PerfFor1Decompress 0 numFrameBits 1 decompressed 104857600 in 1021 msecs, 102700 ints/msec, (25 iters). test901PerfFor1Decompress 1 numFrameBits 1 decompressed 171966464 in 1020 msecs, 168594 ints/msec, (41 iters). test901PerfFor1Decompress 2 numFrameBits 1 decompressed 171966464 in 1012 msecs, 169927 ints/msec, (41 iters). test902PerfFor2Decompress starting Compressed 128 bytes into 12, ratio 0.09375 test902PerfFor2Decompress 0 numFrameBits 2 decompressed 130023424 in 1017 msecs, 127849 ints/msec, (31 iters). test902PerfFor2Decompress 1 numFrameBits 2 decompressed 155189248 in 1000 msecs, 155189 ints/msec, (37 iters). test902PerfFor2Decompress 2 numFrameBits 2 decompressed 159383552 in 1022 msecs, 155952 ints/msec, (38 iters). test903PerfFor3Decompress starting Compressed 128 bytes into 16, ratio 0.125 test903PerfFor3Decompress 0 numFrameBits 3 decompressed 188743680 in 1016 msecs, 185771 ints/msec, (45 iters). test903PerfFor3Decompress 1 numFrameBits 3 decompressed 205520896 in 1018 msecs, 201886 ints/msec, (49 iters). test903PerfFor3Decompress 2 numFrameBits 3 decompressed 201326592 in 1026 msecs, 196224 ints/msec, (48 iters). test904PerfFor4Decompress starting Compressed 128 bytes into 20, ratio 0.15625 test904PerfFor4Decompress 0 numFrameBits 4 decompressed 146800640 in 1014 msecs, 144773 ints/msec, (35 iters). test904PerfFor4Decompress 1 numFrameBits 4 decompressed 159383552 in 1007 msecs, 158275 ints/msec, (38 iters). test904PerfFor4Decompress 2 numFrameBits 4 decompressed 159383552 in 1011 msecs, 157649 ints/msec, (38 iters). test905PerfFor5Decompress starting Compressed 128 bytes into 24, ratio 0.1875 test905PerfFor5Decompress 0 numFrameBits 5 decompressed 159383552 in 1010 msecs, 157805 ints/msec, (38 iters). test905PerfFor5Decompress 1 numFrameBits 5 decompressed 176160768 in 1009 msecs, 174589 ints/msec, (42 iters). test905PerfFor5Decompress 2 numFrameBits 5 decompressed 176160768 in 1004 msecs, 175458 ints/msec, (42 iters). test906PerfFor6Decompress starting Compressed 128 bytes into 28, ratio 0.21875 test906PerfFor6Decompress 0 numFrameBits 6 decompressed 117440512 in 1001 msecs, 117323 ints/msec, (28 iters). test906PerfFor6Decompress 1 numFrameBits 6 decompressed 125829120 in 1002 msecs, 125577 ints/msec, (30 iters). test906PerfFor6Decompress 2 numFrameBits 6 decompressed 125829120 in 1004 msecs, 125327 ints/msec, (30 iters). test907PerfFor7Decompress starting Compressed 128 bytes into 32, ratio 0.25 test907PerfFor7Decompress 0 numFrameBits 7 decompressed 125829120 in 1016 msecs, 123847 ints/msec, (30 iters). test907PerfFor7Decompress 1 numFrameBits 7 decompressed 150994944 in 1015 msecs, 148763 ints/msec, (36 iters). test907PerfFor7Decompress 2 numFrameBits 7 decompressed 150994944 in 1012 msecs, 149204 ints/msec, (36 iters). test908PerfFor8Decompress starting Compressed 124 bytes into 35, ratio 0.28225806 test908PerfFor8Decompress 0 numFrameBits 8 decompressed 85327872 in 1015 msecs, 84066 ints/msec, (21 iters). test908PerfFor8Decompress 1 numFrameBits 8 decompressed 121896960 in 1021 msecs, 119389 ints/msec, (30 iters). test908PerfFor8Decompress 2 numFrameBits 8 decompressed 121896960 in 1022 msecs, 119272 ints/msec, (30 iters). test909PerfFor9Decompress starting Compressed 124 bytes into 39, ratio 0.31451613 test909PerfFor9Decompress 0 numFrameBits 9 decompressed 52822016 in 1016 msecs, 51990 ints/msec, (13 iters). test909PerfFor9Decompress 1 numFrameBits 9 decompressed 56885248 in 1028 msecs, 55335 ints/msec, (14 iters). test909PerfFor9Decompress 2 numFrameBits 9 decompressed 56885248 in 1029 msecs, 55282 ints/msec, (14 iters). {noformat} The loop for 9 bits is not unrolled, unrolling makes it a tiny little bit (55->54 Mint/sec) slower. > PFOR implementation > ------------------- > > Key: LUCENE-1410 > URL: https://issues.apache.org/jira/browse/LUCENE-1410 > Project: Lucene - Java > Issue Type: New Feature > Components: Other > Reporter: Paul Elschot > Priority: Minor > Attachments: LUCENE-1410b.patch, TestPFor2.java, TestPFor2.java > > Original Estimate: 21840h > Remaining Estimate: 21840h > > Implementation of Patched Frame of Reference. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]