jainankitk commented on a change in pull request #563: Support for moving FST
offheap except PK
URL: https://github.com/apache/lucene-solr/pull/563#discussion_r255330316
##########
File path:
lucene/core/src/java/org/apache/lucene/codecs/blocktree/FieldReader.java
##########
@@ -86,9 +88,14 @@
if (indexIn != null) {
final IndexInput clone = indexIn.clone();
- //System.out.println("start=" + indexStartFP + " field=" +
fieldInfo.name);
clone.seek(indexStartFP);
- index = new FST<>(clone, ByteSequenceOutputs.getSingleton());
+ // Initialize FST offheap if index is MMapDirectory and
+ // docCount != sumDocFreq implying field is not primary key
+ if (clone instanceof ByteBufferIndexInput && this.docCount !=
this.sumDocFreq) {
+ index = new FST<>(clone, ByteSequenceOutputs.getSingleton(), new
OffHeapFSTStore());
+ } else {
+ index = new FST<>(clone, ByteSequenceOutputs.getSingleton());
Review comment:
I added print statements while running the benchmarks, and the
classification looks correct:
```
Initializing field offheap start=55 field=Date.taxonomy
Initializing field offheap start=76 field=DayOfYear.sortedset
Initializing field offheap start=97 field=Month.sortedset
Initializing field offheap start=118 field=body
Initializing field onheap start=267 field=date
Initializing field onheap start=289 field=groupend
Initializing field onheap start=311 field=id
Initializing field onheap start=333 field=title
```
Though, when I restricted tests to PKLookups only using
comp.addTaskPattern('PKLookup') in localrun.py, results look as expected:
```
wikimedium10k
TaskQPS baseline StdDevQPS candidate StdDev Pct
diff
PKLookup 163.29 (1.6%) 164.80 (2.1%) 0.9%
(-2% - 4%)
```
```
wikimedium10m
TaskQPS baseline StdDevQPS candidate StdDev Pct diff
PKLookup 114.29 (1.7%) 114.73 (1.2%) 0.4% (
-2% - 3%)
```
I guess we are good then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
