liangkaiwen commented on code in PR #4935:
URL: https://github.com/apache/solr/pull/4935#discussion_r4621390129


##########
solr/core/src/java/org/apache/solr/core/StandardDirectoryFactory.java:
##########
@@ -42,14 +50,77 @@
  * behavior.
  *
  * <p>File based DirectoryFactory implementations generally extend this class.
+ *
+ * <p>Can set the following parameters:
+ *
+ * <ul>
+ *   <li>preload -- Whether to load each index file into the OS page cache 
when that file is opened.
+ *   <li>preloadExtensions -- Comma separated file extensions, such as {@code 
vex,vec}. Only files
+ *       whose name ends with a listed extension are loaded into the OS page 
cache when those files
+ *       are opened. Takes precedence over {@code preload}.
+ * </ul>
+ *
+ * <p>Both parameters only apply when the underlying directory is an {@link 
MMapDirectory}, which is
+ * what {@link FSDirectory#open} selects on most platforms.
  */
 public class StandardDirectoryFactory extends CachingDirectoryFactory {
 
   private static final Logger log = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
 
+  private boolean preload;
+  private Set<String> preloadExtensions = Set.of();
+
+  @Override
+  public void init(NamedList<?> args) {
+    super.init(args);
+    SolrParams params = args.toSolrParams();
+    preload = params.getBool("preload", false); // default turn-off
+    preloadExtensions = 
parsePreloadExtensions(params.get("preloadExtensions"));
+    if (preload && !preloadExtensions.isEmpty()) {
+      log.info(
+          "Ignoring preload=true because preloadExtensions was provided, so 
only loading files with extensions: {}",
+          preloadExtensions);
+    }
+  }
+
+  private static Set<String> parsePreloadExtensions(String value) {
+    if (value == null) {
+      return Set.of();
+    }
+    Set<String> extensions = new LinkedHashSet<>();
+    for (String extension : value.split(",")) {
+      extension = extension.trim().toLowerCase(Locale.ROOT);
+      if (!extension.isEmpty()) {
+        extensions.add(extension.startsWith(".") ? extension : "." + 
extension);
+      }
+    }
+    return Collections.unmodifiableSet(extensions);
+  }
+
+  @VisibleForTesting
+  BiPredicate<String, IOContext> preloadPredicate() {
+    if (preloadExtensions.isEmpty()) {
+      return preload ? MMapDirectory.ALL_FILES : MMapDirectory.NO_FILES;
+    }
+    Set<String> extensions = preloadExtensions;
+    return (name, ioContext) -> {
+      int dot = name.lastIndexOf('.');
+      return dot >= 0 && 
extensions.contains(name.substring(dot).toLowerCase(Locale.ROOT));
+    };
+  }
+
+  /** Applies the configured preload settings, if the given directory supports 
them. */
+  protected void applyPreload(Directory directory) {
+    if (directory instanceof MMapDirectory mMapDirectory) {

Review Comment:
   The subtle difference that you might be missing here is that this is 
checking for MMapDirectory (Lucene) not MMapDirectoryFactory (Solr). Fwiw, I 
also missed this initially and took a lot of digging to wrap my head around. 
From my current understanding, in Lucene world MMapDirectory and 
NRTCachingDirectory do not have any relationship to one another; you can create 
an NRTCachingDirectory that is composed of an MMapDirectory. In fact in Solr, 
this is exactly what NRTCachingDirectoryFactory does - it creates an 
NRTCachingDirectory type for the cache behavior, but underneath it also has the 
MMapDirectory as its underlying FSDirectory behavior. I strongly dislike how 
this is laid out in Solr, and in fact this is probably why there is a bug that 
the existing preload function can only be applied to MMapDirectoryFactory, when 
in reality it could be applied to either factory. Hopefully this explanation 
makes some sense



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to