liangkaiwen commented on code in PR #4935:
URL: https://github.com/apache/solr/pull/4935#discussion_r4619632289


##########
solr/core/src/java/org/apache/solr/core/StandardDirectoryFactory.java:
##########
@@ -42,14 +50,77 @@
  * behavior.
  *
  * <p>File based DirectoryFactory implementations generally extend this class.
+ *
+ * <p>Can set the following parameters:
+ *
+ * <ul>
+ *   <li>preload -- Whether to load each index file into the OS page cache 
when that file is opened.
+ *   <li>preloadExtensions -- Comma separated file extensions, such as {@code 
vex,vec}. Only files
+ *       whose name ends with a listed extension are loaded into the OS page 
cache when those files
+ *       are opened. Takes precedence over {@code preload}.
+ * </ul>
+ *
+ * <p>Both parameters only apply when the underlying directory is an {@link 
MMapDirectory}, which is
+ * what {@link FSDirectory#open} selects on most platforms.
  */
 public class StandardDirectoryFactory extends CachingDirectoryFactory {
 
   private static final Logger log = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
 
+  private boolean preload;
+  private Set<String> preloadExtensions = Set.of();
+
+  @Override
+  public void init(NamedList<?> args) {
+    super.init(args);
+    SolrParams params = args.toSolrParams();
+    preload = params.getBool("preload", false); // default turn-off
+    preloadExtensions = 
parsePreloadExtensions(params.get("preloadExtensions"));
+    if (preload && !preloadExtensions.isEmpty()) {
+      log.info(
+          "Ignoring preload=true because preloadExtensions was provided, so 
only loading files with extensions: {}",
+          preloadExtensions);
+    }
+  }
+
+  private static Set<String> parsePreloadExtensions(String value) {
+    if (value == null) {
+      return Set.of();
+    }
+    Set<String> extensions = new LinkedHashSet<>();

Review Comment:
   I've gone back and forth on whether this feature can have ordered loading. 
My current thought is that we should keep it unordered, and add this 
functionality in a later change if we feel a need for it. Reasons being
   
   1) Lucene code controls the file scanning order, and I believe currently it 
is by segment. We would need some mechanism to scan the full index and order 
the files if we wanted to be true to the ordered behavior
   
   2) We have no current way of knowing when physical memory fills up. When it 
does, the behavior would be to swap rather than to stop, thus making an oddity 
that the correct order would be to load the highest priority files last.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to