mikemccand commented on a change in pull request #640: LUCENE-8671: Introduce
Reader attributes
URL: https://github.com/apache/lucene-solr/pull/640#discussion_r274495805
##########
File path:
lucene/core/src/java/org/apache/lucene/codecs/blocktree/BlockTreeTermsReader.java
##########
@@ -75,6 +74,41 @@
public final class BlockTreeTermsReader extends FieldsProducer {
+ /**
+ * An enum that allows to control if term index FSTs are loaded into memory
or read off-heap
+ */
+ public enum FSTLoadMode {
+ /**
+ * Always read FSTs from disk.
+ * NOTE: If this option is used the FST will be read off-heap even if
buffered directory implementations
+ * are used.
+ */
+ OFF_HEAP,
+ /**
+ * Never read FSTs from disk ie. all fields FSTs are loaded into memory
+ */
+ ON_HEAP,
+ /**
+ * Always read FSTs from disk.
+ * An exception is made for ID fields in an IndexWriter context which are
always loaded into memory.
+ * This is useful to guarantee best update performance even if a non
MMapDirectory is used.
+ * NOTE: If this option is used the FST will be read off-heap even if
buffered directory implementations
+ * are used.
+ * See {@link FSTLoadMode#AUTO}
+ */
+ OPTIMIZE_UPDATES_OFF_HEAP,
+ /**
+ * Automatically make the decision if FSTs are read from disk depending if
the segment read from an MMAPDirectory
+ * An exception is made for ID fields in an IndexWriter context which are
always loaded into memory.
+ */
+ AUTO
+ }
+
+ /** Attribute key for fst mode. */
+ public static final String FST_MODE_KEY = "blocktree.terms.fst.mode";
+ /** Attribute key for the ID field. ID fields are forced on-heap if set **/
+ public static final String ID_FIELD = "blocktree.terms.fst.id_field"; //
TODO should this only happen in a writer context?
Review comment:
It's a little odd to have the notion of an "id field" down this low in
Lucene -- maybe instead we could parameterize the keys by field name? So e.g.
`blocktree.terms.fst.$field` would be set to the `FSTLoadMode` enum values?
This would also give users per-field control.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
