li-afaris commented on a change in pull request #1500:
URL: https://github.com/apache/samza/pull/1500#discussion_r631495076
##########
File path:
samza-kv-rocksdb/src/main/java/org/apache/samza/storage/kv/RocksDbOptionsHelper.java
##########
@@ -109,6 +111,8 @@ public static Options options(Config storeConfig, int
numTasksForContainer, File
options.setMaxLogFileSize(storeConfig.getLong(ROCKSDB_MAX_LOG_FILE_SIZE_BYTES,
64 * 1024 * 1024L));
options.setKeepLogFileNum(storeConfig.getLong(ROCKSDB_KEEP_LOG_FILE_NUM,
2));
options.setDeleteObsoleteFilesPeriodMicros(storeConfig.getLong(ROCKSDB_DELETE_OBSOLETE_FILES_PERIOD_MICROS,
21600000000L));
+ options.setMaxOpenFiles(storeConfig.getInt(ROCKSDB_MAX_OPEN_FILES, -1));
+
options.setMaxOpenFiles(storeConfig.getInt(ROCKSDB_MAX_FILE_OPENING_THREADS,
16));
Review comment:
This is not a typo. We want to limit the number of sst files which are
opened to prevent exceeding ulimit values.
Here's the java doc description for the method
```
Number of open files that can be used by the DB. You may need to increase
this if your database has a large working set. Value -1 means files opened are
always kept open. You can estimate number of files based on
target_file_size_base and target_file_size_multiplier for level-based
compaction. For universal-style compaction, you can usually set it to -1.
```
https://javadoc.io/doc/org.rocksdb/rocksdbjni/6.6.4/org/rocksdb/Options.html#maxOpenFiles--
Which matches this RocksDB option
```
// Number of open files that can be used by the DB. You may need to
// increase this if your database has a large working set. Value -1 means
// files opened are always kept open. You can estimate number of files
based
// on target_file_size_base and target_file_size_multiplier for level-based
// compaction. For universal-style compaction, you can usually set it to
-1.
```
https://github.com/facebook/rocksdb/blob/master/include/rocksdb/options.h#L460
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
