[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1480: MINIFICPP-2007 Add compression options for flowfile and content repo

[via GitHub]

lordgamez commented on code in PR #1480:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1480#discussion_r1108469672


##
conf/minifi.properties:
##
@@ -29,9 +29,11 @@ nifi.provenance.repository.max.storage.time=1 MIN
 nifi.provenance.repository.max.storage.size=1 MB
 nifi.flowfile.repository.directory.default=${MINIFI_HOME}/flowfile_repository
 nifi.flowfile.checkpoint.directory.default=${MINIFI_HOME}/flowfile_checkpoint
+# nifi.flowfile.repository.rocksdb.compression=auto

Review Comment:
   I tried switching compression types between restarts and the flow continued 
where it left off previously. I could see in the rocksdb log files the used 
compression was changed to the new one in the minifi property file so it seems 
to me that rocksdb could handle the config change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1480: MINIFICPP-2007 Add compression options for flowfile and content repo

[via GitHub]

lordgamez commented on code in PR #1480:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1480#discussion_r1100228433


##
extensions/rocksdb-repos/FlowFileRepository.h:
##
@@ -131,11 +131,35 @@ class FlowFileRepository : public ThreadedRepository, 
public SwapManager {
 // To avoid DB write issues during heavy load it's recommended to have 
high number of buffer.
 // Rocksdb's stall feature can also trigger in case the number of buffers 
is >= 3.
 // The more buffers we have the more memory rocksdb can utilize without 
significant memory consumption under low load.
-auto cf_options = [] (rocksdb::ColumnFamilyOptions& cf_opts) {
+auto cf_options = [] (rocksdb::ColumnFamilyOptions& cf_opts) {
   cf_opts.OptimizeForPointLookup(4);
   cf_opts.write_buffer_size = 8ULL << 20U;
   cf_opts.max_write_buffer_number = 20;
   cf_opts.min_write_buffer_number_to_merge = 1;
+  std::string value;
+  if (configure->get(Configure::nifi_flow_repository_rocksdb_compression, 
value) && !value.empty()) {
+#ifdef WIN32
+if (value == "xpress") {
+  cf_opts.compression = rocksdb::CompressionType::kXpressCompression;
+} else {
+  throw Exception(REPOSITORY_EXCEPTION, "RocksDB compression type not 
supported: " + value);
+}
+#else
+if (value == "zlib") {
+  cf_opts.compression = rocksdb::CompressionType::kZlibCompression;
+} else if (value == "bzip2") {
+  cf_opts.compression = rocksdb::CompressionType::kBZip2Compression;

Review Comment:
   Great idea! Updated in 1d51fe6f83e37c7f9ae81306f55d920abc1f69e1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org