This is an automated email from the ASF dual-hosted git repository.
bharathkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/samza.git
The following commit(s) were added to refs/heads/master by this push:
new 7e33b96 SAMZA-2267: Allow overriding RocksDB file size parameters
7e33b96 is described below
commit 7e33b96fe8c8ea596e729643a0535924de6c4f3d
Author: omkardeshpande8 <[email protected]>
AuthorDate: Thu Jul 11 17:52:24 2019 -0700
SAMZA-2267: Allow overriding RocksDB file size parameters
Author: odeshpande <Intuit03>
Reviewers: mynameborat <[email protected]>
Closes #1099 from omkardeshpande8/rocksdb-config-improvement and squashes
the following commits:
4cf77ba3 [odeshpande] reverted changes to configuration-table.html, since
it is deprecated
cafb46f3 [odeshpande] Updated documentation in samza-configurations.md and
added a check for presence of max manifest file size property
c9cb8846 [odeshpande] Updated documentation in samza-configurations.md and
added a check for presence of max manifest file size property
b136ee71 [odeshpande] Allowing two additional rocksdb configuration
properties to be overriden
---
docs/learn/documentation/versioned/jobs/samza-configurations.md | 2 ++
.../java/org/apache/samza/storage/kv/RocksDbOptionsHelper.java | 8 +++++++-
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/docs/learn/documentation/versioned/jobs/samza-configurations.md
b/docs/learn/documentation/versioned/jobs/samza-configurations.md
index 1e702b0..3f12594 100644
--- a/docs/learn/documentation/versioned/jobs/samza-configurations.md
+++ b/docs/learn/documentation/versioned/jobs/samza-configurations.md
@@ -274,6 +274,8 @@ These properties define Samza's storage mechanism for
efficient [stateful stream
|stores.**_store-name_**.<br>rocksdb.max.log.file.size.bytes|67108864|The
maximum size in bytes of the RocksDB LOG file before it is rotated.|
|stores.**_store-name_**.<br>rocksdb.keep.log.file.num|2|The number of RocksDB
LOG files (including rotated LOG.old.* files) to keep.|
|stores.**_store-name_**.<br>rocksdb.metrics.list|(none)|A list of [RocksDB
properties](https://github.com/facebook/rocksdb/blob/master/include/rocksdb/db.h#L409)
to expose as metrics (gauges).|
+|stores.**_store-name_**.<br>rocksdb.delete.obsolete.files.period.micros|21600000000|This
property specifies the period in microseconds to delete obsolete files
regardless of files removed during compaction. Allowed range is up to
9223372036854775807.|
+|stores.**_store-name_**.<br>rocksdb.max.manifest.file.size|18446744073709551615|This
property specifies the maximum size of the MANIFEST data file, after which it
is rotated. Default value is also the maximum, making it practically unlimited:
only one manifest file is used.|
### <a name="deployment"></a>[5. Deployment](#deployment)
Samza supports both standalone and clustered ([YARN](yarn-jobs.html))
[deployment models](../deployment/deployment-model.html). Below are the
configurations options for both models.
diff --git
a/samza-kv-rocksdb/src/main/java/org/apache/samza/storage/kv/RocksDbOptionsHelper.java
b/samza-kv-rocksdb/src/main/java/org/apache/samza/storage/kv/RocksDbOptionsHelper.java
index 10052c6..aacf848 100644
---
a/samza-kv-rocksdb/src/main/java/org/apache/samza/storage/kv/RocksDbOptionsHelper.java
+++
b/samza-kv-rocksdb/src/main/java/org/apache/samza/storage/kv/RocksDbOptionsHelper.java
@@ -42,6 +42,8 @@ public class RocksDbOptionsHelper {
private static final String ROCKSDB_NUM_WRITE_BUFFERS =
"rocksdb.num.write.buffers";
private static final String ROCKSDB_MAX_LOG_FILE_SIZE_BYTES =
"rocksdb.max.log.file.size.bytes";
private static final String ROCKSDB_KEEP_LOG_FILE_NUM =
"rocksdb.keep.log.file.num";
+ private static final String ROCKSDB_DELETE_OBSOLETE_FILES_PERIOD_MICROS =
"rocksdb.delete.obsolete.files.period.micros";
+ private static final String ROCKSDB_MAX_MANIFEST_FILE_SIZE =
"rocksdb.max.manifest.file.size";
public static Options options(Config storeConfig, int numTasksForContainer,
File storeDir, StorageEngineFactory.StoreMode storeMode) {
Options options = new Options();
@@ -106,7 +108,11 @@ public class RocksDbOptionsHelper {
options.setMaxLogFileSize(storeConfig.getLong(ROCKSDB_MAX_LOG_FILE_SIZE_BYTES,
64 * 1024 * 1024L));
options.setKeepLogFileNum(storeConfig.getLong(ROCKSDB_KEEP_LOG_FILE_NUM,
2));
-
+
options.setDeleteObsoleteFilesPeriodMicros(storeConfig.getLong(ROCKSDB_DELETE_OBSOLETE_FILES_PERIOD_MICROS,
21600000000L));
+ // The default for rocksdb is 18446744073709551615, which is larger than
java Long.MAX_VALUE. Hence setting it only if it's passed.
+ if(storeConfig.containsKey(ROCKSDB_MAX_MANIFEST_FILE_SIZE)) {
+
options.setMaxManifestFileSize(storeConfig.getLong(ROCKSDB_MAX_MANIFEST_FILE_SIZE));
+ }
// use prepareForBulk load only when i. the store is being requested in
BulkLoad mode
// and ii. the storeDirectory does not exist (fresh restore), because bulk
load does not work seamlessly with
// existing stores : https://github.com/facebook/rocksdb/issues/2734
