[jira] [Commented] (LUCENE-8932) Allow BKDReader packedIndex to be off heap
[ https://issues.apache.org/jira/browse/LUCENE-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16924293#comment-16924293 ] Jack Conradson commented on LUCENE-8932: [~jpountz] Thanks for the thorough review! I will incorporate your feedback into a patch next week and try to move forward from there. > Allow BKDReader packedIndex to be off heap > -- > > Key: LUCENE-8932 > URL: https://issues.apache.org/jira/browse/LUCENE-8932 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jack Conradson >Priority: Minor > Attachments: LUCENE-8932.patch > > > This change modifies BKDReader to read the packedIndex bytes off heap rather > than load them all on heap at a single time. > Questions for discussion: > # Should BKDReader only support packedIndex off heap? > # If not, how should the choice be made? > Using luceneutils IndexAndSearchOpenStreetMaps present the following test > results: > with -box -points (patch) > READER MB: 1.1345596313476562 > BEST M hits/sec: 73.34277344984474 > BEST QPS: 74.63011169783009 > with -box -points (original) > READER MB: 1.7249317169189453 > BEST M hits/sec: 73.77125157623486 > BEST QPS: 75.06611062353801 > with -nearest 10 -points (patch) > READER MB: 1.1345596313476562 > BEST M hits/sec: 0.013586298373879497 > BEST QPS: 1358.6298373879497 > with -nearest 10 -points (original) > READER MB: 1.7249317169189453 > BEST M hits/sec: 0.01445208197367343 > BEST QPS: 1445.208197367343 > with -box -geo3d (patch) > READER MB: 1.1345596313476562 > BEST M hits/sec: 39.84968715299074 > BEST QPS: 40.54914292796736 > with -box -geo3d (original) > READER MB: 1.7456226348876953 > BEST M hits/sec: 40.45051734329004 > BEST QPS: 41.160519101846695 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8932) Allow BKDReader packedIndex to be off heap
[ https://issues.apache.org/jira/browse/LUCENE-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923137#comment-16923137 ] Adrien Grand commented on LUCENE-8932: -- I was hoping LUCENE-8833 would land and we could build on it but it's taking some more time than I thought yet I don't think we need to wait. I'd suggest the following to move forward: - Add a parameter to BKDReader's constructor to decide whether the index should be on or off heap. This is useful so that tests can use both flavors of index. - Keep a constructor that only takes an IndexInput, and load the index off heap iff the IndexInput extends ByteBufferIndexInput. This would be consistent with the terms index. - Update TestBKD to randomly load the index on or off-heap. Then when LUCENE-8833 lands we can look into doing this a bit differently. [~jdconradson] Does it sound good to you? The patch looks good to me in general. One minor thing that looks inconsistent to me is that BKDInput#setPosition doesn't throw an IOException. It forces the offheap impl to catch and rethrow, while readBytes throws an IOException so it's up to callers to deal with the exception. In my opinion, we should make setPosition and readBytes consistent: either both throw an IOException, or none of them do? > Allow BKDReader packedIndex to be off heap > -- > > Key: LUCENE-8932 > URL: https://issues.apache.org/jira/browse/LUCENE-8932 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jack Conradson >Priority: Minor > Attachments: LUCENE-8932.patch > > > This change modifies BKDReader to read the packedIndex bytes off heap rather > than load them all on heap at a single time. > Questions for discussion: > # Should BKDReader only support packedIndex off heap? > # If not, how should the choice be made? > Using luceneutils IndexAndSearchOpenStreetMaps present the following test > results: > with -box -points (patch) > READER MB: 1.1345596313476562 > BEST M hits/sec: 73.34277344984474 > BEST QPS: 74.63011169783009 > with -box -points (original) > READER MB: 1.7249317169189453 > BEST M hits/sec: 73.77125157623486 > BEST QPS: 75.06611062353801 > with -nearest 10 -points (patch) > READER MB: 1.1345596313476562 > BEST M hits/sec: 0.013586298373879497 > BEST QPS: 1358.6298373879497 > with -nearest 10 -points (original) > READER MB: 1.7249317169189453 > BEST M hits/sec: 0.01445208197367343 > BEST QPS: 1445.208197367343 > with -box -geo3d (patch) > READER MB: 1.1345596313476562 > BEST M hits/sec: 39.84968715299074 > BEST QPS: 40.54914292796736 > with -box -geo3d (original) > READER MB: 1.7456226348876953 > BEST M hits/sec: 40.45051734329004 > BEST QPS: 41.160519101846695 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8932) Allow BKDReader packedIndex to be off heap
[ https://issues.apache.org/jira/browse/LUCENE-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892193#comment-16892193 ] Adrien Grand commented on LUCENE-8932: -- This is interesting! It doesn't seem to affect performance at all so I believe we could just load the index off-heap all the time? And maybe something like LUCENE-8833 could help make sure it is hot in the page cache. > Allow BKDReader packedIndex to be off heap > -- > > Key: LUCENE-8932 > URL: https://issues.apache.org/jira/browse/LUCENE-8932 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jack Conradson >Priority: Minor > Attachments: LUCENE-8932.patch > > > This change modifies BKDReader to read the packedIndex bytes off heap rather > than load them all on heap at a single time. > Questions for discussion: > # Should BKDReader only support packedIndex off heap? > # If not, how should the choice be made? > Using luceneutils IndexAndSearchOpenStreetMaps present the following test > results: > with -box -points (patch) > READER MB: 1.1345596313476562 > BEST M hits/sec: 73.34277344984474 > BEST QPS: 74.63011169783009 > with -box -points (original) > READER MB: 1.7249317169189453 > BEST M hits/sec: 73.77125157623486 > BEST QPS: 75.06611062353801 > with -nearest 10 -points (patch) > READER MB: 1.1345596313476562 > BEST M hits/sec: 0.013586298373879497 > BEST QPS: 1358.6298373879497 > with -nearest 10 -points (original) > READER MB: 1.7249317169189453 > BEST M hits/sec: 0.01445208197367343 > BEST QPS: 1445.208197367343 > with -box -geo3d (patch) > READER MB: 1.1345596313476562 > BEST M hits/sec: 39.84968715299074 > BEST QPS: 40.54914292796736 > with -box -geo3d (original) > READER MB: 1.7456226348876953 > BEST M hits/sec: 40.45051734329004 > BEST QPS: 41.160519101846695 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
