[
https://issues.apache.org/jira/browse/LUCENE-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-1410:
---------------------------------------
Attachment: TestPFor2.java
New TestPFor2 attached. I changed vInt -> int in the prints and now
compute "best" bit size per-block using getNumFrameBits() and use that
per block.
I took a big frq file and computed %tg for each #bits:
||bits||count||pct||
|1|0|0.0|
|2|24328|0.9|
|3|94689|3.7|
|4|213887|8.4|
|5|277510|10.8|
|6|284905|11.1|
|7|193081|7.5|
|8|262857|10.3|
|9|260460|10.2|
|10|212046|8.3|
|11|162872|6.4|
|12|109801|4.3|
|13|77366|3.0|
|14|56620|2.2|
|15|34294|1.3|
|16|31000|1.2|
|17|28803|1.1|
|18|21617|0.8|
|19|22317|0.9|
|20|30326|1.2|
|21|162663|6.4|
And for prx:
||bits||count||pct||
|1|23034|0.7|
|2|12637|0.4|
|3|49286|1.4|
|4|56344|1.6|
|5|69385|2.0|
|6|81964|2.4|
|7|108847|3.1|
|8|179296|5.2|
|9|428787|12.4|
|10|873828|25.2|
|11|957913|27.7|
|12|534426|15.4|
|13|81856|2.4|
|14|2436|0.1|
|15|474|0.0|
|16|200|0.0|
|17|43|0.0|
|18|17|0.0|
|19|1|0.0|
It's interesting that prx is so much more tightly clustered than frq.
New results for decoding vInt vs pfor, for frq file:
{code}
327864576 ints; 431137397 bytes compressed vs orig size 447077047
decompress using pfor:
2514 msec to decode 327864576 ints (130415 ints/msec)
2171 msec to decode 327864576 ints (151020 ints/msec)
2137 msec to decode 327864576 ints (153422 ints/msec)
2148 msec to decode 327864576 ints (152637 ints/msec)
2067 msec to decode 327864576 ints (158618 ints/msec)
decompress using readVInt:
4549 msec to read 327864691 ints (72074 ints/msec)
4421 msec to read 327864691 ints (74160 ints/msec)
3240 msec to read 327864691 ints (101192 ints/msec)
3199 msec to read 327864691 ints (102489 ints/msec)
3323 msec to read 327864691 ints (98665 ints/msec)
PFor is 54.766% faster
{code}
and prx file:
{code}
encode _l.prx to _l.prx.pfor...
442979072 ints; 628580878 bytes compressed vs orig size 651670377
decompress using pfor:
6385 msec to decode 442979072 ints (69378 ints/msec)
5904 msec to decode 442979072 ints (75030 ints/msec)
5796 msec to decode 442979072 ints (76428 ints/msec)
5807 msec to decode 442979072 ints (76283 ints/msec)
5803 msec to decode 442979072 ints (76336 ints/msec)
decompress using readVInt:
6893 msec to read 442979104 ints (64265 ints/msec)
6759 msec to read 442979104 ints (65539 ints/msec)
5304 msec to read 442979104 ints (83517 ints/msec)
5275 msec to read 442979104 ints (83977 ints/msec)
5792 msec to read 442979104 ints (76481 ints/msec)
PFor is 8.989% slower
{code}
Some comments:
* In both cases, the pfor file a bit smaller than the original
Lucene file. And, it's unfair because I inject the 4 extra bytes
per block (which I think won't be needed "for real").
* For frq file, pfor is quite a bit faster (54.8%) but for prx file
it's slower (9.0%). Maybe it's because prx file has much higher
occurrence of bigger bit sizes (where we don't have unrolled
version)? It's odd; maybe I'm doing something wrong. With a
fixed 6 bits it is quite a bit faster (I retested to verify!)
which is also weird because there would be many exceptions.
Exception processing ought to be slower than non-exception
processing!
Stepping back a bit: I wonder what %tg of the time in a "typical"
Lucene search is spent decoding vInts? That would bound how much
improvement we could ever expect from this optimization during
searching.
> PFOR implementation
> -------------------
>
> Key: LUCENE-1410
> URL: https://issues.apache.org/jira/browse/LUCENE-1410
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Other
> Reporter: Paul Elschot
> Priority: Minor
> Attachments: LUCENE-1410b.patch, TestPFor2.java, TestPFor2.java
>
> Original Estimate: 21840h
> Remaining Estimate: 21840h
>
> Implementation of Patched Frame of Reference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
