[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1480: MINIFICPP-2007 Add compression options for flowfile and content repo

[via GitHub]

szaszm commented on code in PR #1480:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1480#discussion_r1108397217


##
conf/minifi.properties:
##
@@ -29,9 +29,11 @@ nifi.provenance.repository.max.storage.time=1 MIN
 nifi.provenance.repository.max.storage.size=1 MB
 nifi.flowfile.repository.directory.default=${MINIFI_HOME}/flowfile_repository
 nifi.flowfile.checkpoint.directory.default=${MINIFI_HOME}/flowfile_checkpoint
+# nifi.flowfile.repository.rocksdb.compression=auto

Review Comment:
   Is it possible to change these settings without losing existing repository 
data? So basically does it detect existing data in a different format and do 
conversion at startup? If not, is this something we could make work, possibly 
in a followup?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1480: MINIFICPP-2007 Add compression options for flowfile and content repo

[via GitHub]

szaszm commented on code in PR #1480:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1480#discussion_r1100041964


##
extensions/rocksdb-repos/FlowFileRepository.h:
##
@@ -131,11 +131,35 @@ class FlowFileRepository : public ThreadedRepository, 
public SwapManager {
 // To avoid DB write issues during heavy load it's recommended to have 
high number of buffer.
 // Rocksdb's stall feature can also trigger in case the number of buffers 
is >= 3.
 // The more buffers we have the more memory rocksdb can utilize without 
significant memory consumption under low load.
-auto cf_options = [] (rocksdb::ColumnFamilyOptions& cf_opts) {
+auto cf_options = [] (rocksdb::ColumnFamilyOptions& cf_opts) {
   cf_opts.OptimizeForPointLookup(4);
   cf_opts.write_buffer_size = 8ULL << 20U;
   cf_opts.max_write_buffer_number = 20;
   cf_opts.min_write_buffer_number_to_merge = 1;
+  std::string value;
+  if (configure->get(Configure::nifi_flow_repository_rocksdb_compression, 
value) && !value.empty()) {
+#ifdef WIN32
+if (value == "xpress") {
+  cf_opts.compression = rocksdb::CompressionType::kXpressCompression;
+} else {
+  throw Exception(REPOSITORY_EXCEPTION, "RocksDB compression type not 
supported: " + value);
+}
+#else
+if (value == "zlib") {
+  cf_opts.compression = rocksdb::CompressionType::kZlibCompression;
+} else if (value == "bzip2") {
+  cf_opts.compression = rocksdb::CompressionType::kBZip2Compression;

Review Comment:
   I propose to add an option like "auto" or "yes", which chooses a good 
default compression algorithm on all platforms. This way we can keep the config 
platform-independent, and people don't need to become experts in compression 
algorithms to make a good choice.
   On Windows, it obviously has to be xpress, since there's nothing else. On 
others, I'd go for zstd.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org