alessandrobenedetti commented on code in PR #4935:
URL: https://github.com/apache/solr/pull/4935#discussion_r4482916218
##########
solr/core/src/java/org/apache/solr/core/StandardDirectoryFactory.java:
##########
@@ -42,14 +50,77 @@
* behavior.
*
* <p>File based DirectoryFactory implementations generally extend this class.
+ *
+ * <p>Can set the following parameters:
+ *
+ * <ul>
+ * <li>preload -- Whether to load each index file into the OS page cache
when that file is opened.
+ * <li>preloadExtensions -- Comma separated file extensions, such as {@code
vex,vec}. Only files
+ * whose name ends with a listed extension are loaded into the OS page
cache when those files
+ * are opened. Takes precedence over {@code preload}.
+ * </ul>
+ *
+ * <p>Both parameters only apply when the underlying directory is an {@link
MMapDirectory}, which is
+ * what {@link FSDirectory#open} selects on most platforms.
*/
public class StandardDirectoryFactory extends CachingDirectoryFactory {
private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private boolean preload;
+ private Set<String> preloadExtensions = Set.of();
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ SolrParams params = args.toSolrParams();
+ preload = params.getBool("preload", false); // default turn-off
+ preloadExtensions =
parsePreloadExtensions(params.get("preloadExtensions"));
+ if (preload && !preloadExtensions.isEmpty()) {
+ log.info(
+ "Ignoring preload=true because preloadExtensions was provided, so
only loading files with extensions: {}",
+ preloadExtensions);
+ }
+ }
+
+ private static Set<String> parsePreloadExtensions(String value) {
+ if (value == null) {
+ return Set.of();
+ }
+ Set<String> extensions = new LinkedHashSet<>();
Review Comment:
For @liangkaiwen , do we want order to matter? i.e. we are assigning a
priority to the extensions, so they should be loaded in that order up to the
point RAM is exhausted if exhausted?
If the order is supported at preloading time, then there may be value in
having the extensions ordered after parsing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
