This is an automated email from the ASF dual-hosted git repository.
devriesb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new 5f69648 NIFI-4775: Updating documentation and updating rocksdb version
5f69648 is described below
commit 5f696482b06b868c88a96cf6666e21642dea386b
Author: Brandon Devries <[email protected]>
AuthorDate: Tue Aug 13 14:45:00 2019 -0400
NIFI-4775: Updating documentation and updating rocksdb version
NIFI-4775: Rolling back fix for NIFI-6416
NIFI-4775: Documentation improvements
+1 from markobean
This closes #3648
Signed-off-by: Brandon <[email protected]>
---
nifi-commons/nifi-rocksdb-utils/pom.xml | 2 +-
.../src/main/asciidoc/administration-guide.adoc | 120 ++++++++++++++++++++-
.../repository/WriteAheadFlowFileRepository.java | 2 +-
3 files changed, 120 insertions(+), 4 deletions(-)
diff --git a/nifi-commons/nifi-rocksdb-utils/pom.xml
b/nifi-commons/nifi-rocksdb-utils/pom.xml
index 02ece68..ce42709 100644
--- a/nifi-commons/nifi-rocksdb-utils/pom.xml
+++ b/nifi-commons/nifi-rocksdb-utils/pom.xml
@@ -38,7 +38,7 @@
<dependency>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
- <version>6.0.1</version>
+ <version>6.2.2</version>
</dependency>
</dependencies>
</project>
diff --git a/nifi-docs/src/main/asciidoc/administration-guide.adoc
b/nifi-docs/src/main/asciidoc/administration-guide.adoc
index 25a9769..90b7101 100644
--- a/nifi-docs/src/main/asciidoc/administration-guide.adoc
+++ b/nifi-docs/src/main/asciidoc/administration-guide.adoc
@@ -2453,16 +2453,26 @@ The H2 Settings section defines the settings for the H2
database, which keeps tr
|`nifi.h2.url.append`|This property specifies additional arguments to add to
the connection string for the H2 database. The default value should be used and
should not be changed. It is:
`;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE`.
|====
-
=== FlowFile Repository
The FlowFile repository keeps track of the attributes and current state of
each FlowFile in the system. By default,
this repository is installed in the same root installation directory as all
the other repositories; however, it is advisable
to configure it on a separate drive if available.
+There are currently three implementations of the FlowFile Repository, which
are detailed below.
+
+|====
+|*Property*|*Description*
+|`nifi.flowfile.repository.implementation`|The FlowFile Repository
implementation. The default value is
`org.apache.nifi.controller.repository.WriteAheadFlowFileRepository`. The
other current options are
`org.apache.nifi.controller.repository.VolatileFlowFileRepository` and
`org.apache.nifi.controller.repository.RocksDBFlowFileRepository`.
+|====
+NOTE: Switching repository implementations should only be done on an instance
with zero queued FlowFiles, and should only be done with caution.
+
+=== Write Ahead FlowFile Repository
+
+WriteAheadFlowFileRepository is the default implementation. It persists
FlowFiles to disk, and can optionally be configured to synchronize all changes
to disk. This is very expensive and can significantly reduce NiFi performance.
However, if it is `false`, there could be the potential for data loss if either
there is a sudden power loss or the operating system crashes. The default value
is `false`.
+
|====
|*Property*|*Description*
-|`nifi.flowfile.repository.implementation`|The FlowFile Repository
implementation. The default value is
`org.apache.nifi.controller.repository.WriteAheadFlowFileRepository` and should
only be changed with caution. To store flowfiles in memory instead of on disk
(accepting data loss in the event of power/machine failure or a restart of
NiFi), set this property to
`org.apache.nifi.controller.repository.VolatileFlowFileRepository`.
|`nifi.flowfile.repository.wal.implementation`|If the repository
implementation is configured to use the `WriteAheadFlowFileRepository`, this
property can be used to specify which implementation of the
Write-Ahead Log should be used. The default value is
`org.apache.nifi.wali.SequentialAccessWriteAheadLog`. This version of the
write-ahead log was added in version 1.6.0 of Apache NiFi and was developed
in order to address an issue that exists in the older implementation. In the
event of power loss or an operating system crash, the old implementation was
susceptible to recovering FlowFiles
@@ -2477,6 +2487,112 @@ implementation.
|`nifi.flowfile.repository.always.sync`|If set to `true`, any change to the
repository will be synchronized to the disk, meaning that NiFi will ask the
operating system not to cache the information. This is very expensive and can
significantly reduce NiFi performance. However, if it is `false`, there could
be the potential for data loss if either there is a sudden power loss or the
operating system crashes. The default value is `false`.
|====
+
+=== Volatile FlowFile Repository
+
+This implementation stores FlowFiles in memory instead of on disk. It *will*
result in data loss in the event of power/machine failure or a restart of NiFi.
To use this implementation, set `nifi.flowfile.repository.implementation` to
`org.apache.nifi.controller.repository.VolatileFlowFileRepository`.
+
+
+=== RocksDB FlowFile Repository
+
+This implementation makes use of the RocksDB key-value store. It uses
periodic synchronization to ensure that no created or received data is lost (as
long as `nifi.flowfile.repository.rocksdb.accept.data.loss` is set `false`).
In the event of a failure (e.g. power loss), _work_ done on FlowFiles through
the system (i.e. routing and transformation) may still be lost. Specifically,
the record of these actions may be lost, reverting the affected FlowFiles to a
previous, valid state. Fro [...]
+
+The configuration parameters for this repository fall in to two categories,
"NiFi-centric" and "RocksDB-centric". The NiFi-centric settings have to do
with the operations of the FlowFile Repository and its interaction with NiFi.
The RocksDB-centric settings directly correlate to settings on the underlying
RocksDB repo. More information on these settings can be found in the RocksDB
documentation: https://github.com/facebook/rocksdb/wiki/RocksJava-Basics.
+
+NOTE: Windows users will need to ensure "Microsoft Visual C++ 2015
Redistributable" is installed for this repository to work. See the following
link for more details:
https://github.com/facebook/rocksdb/wiki/RocksJava-Basics#maven-windows.
+
+
+To use this implementation, set `nifi.flowfile.repository.implementation` to
`org.apache.nifi.controller.repository.RocksDBFlowFileRepository`.
+
+*NiFi-centric Configuration Properties*:
+|====
+|*Property*|*Description*
+|`nifi.flowfile.repository.directory`|The location of the FlowFile Repository.
+The default value is`./flowfile_repository`.
+|`nifi.flowfile.repository.rocksdb.sync.warning.period`
+|How often to log warnings if unable to sync.
+The default value is 30 seconds.
+|`nifi.flowfile.repository.rocksdb.claim.cleanup.period`
+|How often to mark content claims destructible (so they can be removed from
the content repo).
+The default value is 30 seconds.
+|`nifi.flowfile.repository.rocksdb.deserialization.threads`
+|How many threads to use on startup restoring the FlowFile state.
+The default value is 16.
+|`nifi.flowfile.repository.rocksdb.deserialization.buffer.size`
+|Size of the buffer to use on startup restoring the FlowFile state.
+The default value is 1000.
+|`nifi.flowfile.repository.rocksdb.sync.period`
+|Frequency at which to force a sync to disk. This is the maximum period a
data creation operation may block if
`nifi.flowfile.repository.rocksdb.accept.data.loss` is `false`.
+The default value is 10 milliseconds.
+|`nifi.flowfile.repository.rocksdb.accept.data.loss`
+|Whether to accept the loss of received / created data. Setting this `true`
increases throughput if loss of data is acceptable.
+The default value is false.
+|`nifi.flowfile.repository.rocksdb.enable.stall.stop`
+|Whether to enable the stall / stop of writes to the repository based on
configured limits. Enabling this feature allows the system to protect itself
by restricting (delaying or denying) operations that increase the total
FlowFile count on the node to prevent the system from being overwhelmed.
+The default value is false.
+|`nifi.flowfile.repository.rocksdb.stall.period`
+|The period of time to stall when the specified criteria are encountered.
+The default value is 100 milliseconds.
+|`nifi.flowfile.repository.rocksdb.stall.flowfile.count`
+|The FlowFile count at which to begin stalling writes to the repo.
+The default value is 800000.
+|`nifi.flowfile.repository.rocksdb.stall.heap.usage.percent`
+|The heap usage at which to begin stalling writes to the repo.
+The default value is 95%.
+|`nifi.flowfile.repository.rocksdb.stop.flowfile.count`
+|The FlowFile count at which to begin stopping the creation of new FlowFiles.
+The default value is 1100000.
+|`nifi.flowfile.repository.rocksdb.stop.heap.usage.percent`
+|The heap usage at which to begin stopping the creation of new FlowFiles.
+The default value is 99.9%.
+|`nifi.flowfile.repository.rocksdb.remove.orphaned.flowfiles.on.startup`
+|Whether to allow the repository to remove FlowFiles it cannot identify on
startup. As this is often the result of a configuration or synchronization
error, it is disabled by default. This should only be enabled if you are
absolutely certain you want to lose the data in question.
+The default value is false.
+|`nifi.flowfile.repository.rocksdb.enable.recovery.mode`
+|Whether to enable "recovery mode". This limits the number of FlowFiles
loaded into the graph at a time, while not actually removing any FlowFiles (or
content) from the system. This allows for the recovery of a system that is
encountering OutOfMemory errors or similar on startup. This should not be
enabled unless necessary to recover a system, and should be disabled as soon as
that has been accomplished.
+
+*WARNING:* While in recovery mode, *do not* make modifications to the graph.
Changes to the graph may result in the inability to restore further FlowFiles
from the repository.
+The default value is false.
+|`nifi.flowfile.repository.rocksdb.recovery.mode.flowfile.count`
+|The number of FlowFiles to load into the graph when in "recovery mode". As
FlowFiles leave the system, additional FlowFiles will be loaded up to this
limit. This setting does not prevent FlowFiles from coming into the system via
normal means.
+The default value is 5000.
+|====
+
+*RocksDB-centric Configuration Properties:*
+|====
+|*Property*|*Description*
+|`nifi.flowfile.repository.rocksdb.parallel.threads`
+|The number of threads to use for flush and compaction. A good value is the
number of cores. See RockDB `DBOptions.setIncreaseParallelism()` for more
information.
+The default value is 8.
+|`nifi.flowfile.repository.rocksdb.max.write.buffer.number`
+|The maximum number of write buffers that are built up in memory. See RockDB
`ColumnFamilyOptions.setMaxWriteBufferNumber()` / `max_write_buffer_number` for
more information.
+The default value is 4.
+|`nifi.flowfile.repository.rocksdb.write.buffer.size`
+|The amount of data to build up in memory before converting to a sorted on
disk file. Larger values increase performance, especially during bulk loads.
Up to `max_write_buffer_number` write buffers may be held in memory at the same
time, so you may wish to adjust this parameter to control memory usage. See
RockDB `ColumnFamilyOptions.setWriteBufferSize()` / `write_buffer_size` for
more information.
+The default value is 256 MB.
+|`nifi.flowfile.repository.rocksdb.level.0.slowdown.writes.trigger`
+|A soft limit on number of level-0 files. Writes are slowed at this point. A
values less than 0 means no write slow down will be triggered by the number of
files in level-0. See RocksDB
`ColumnFamilyOptions.setLevel0SlowdownWritesTrigger()` /
`level0_slowdown_writes_trigger` for more information.
+The default value is 20.
+|`nifi.flowfile.repository.rocksdb.level.0.stop.writes.trigger`
+|The maximum number of level-0 files. Writes will be stopped at this point.
See RocksDB `ColumnFamilyOptions.setLevel0StopWritesTrigger()` /
`level0_stop_writes_trigger` for more information.
+The default value is 40.
+|`nifi.flowfile.repository.rocksdb.delayed.write.bytes.per.second`
+|The limited write rate to the DB if slowdown is triggered. RocksDB may
decide to slow down more if the compaction gets behind further. See RocksDB
`DBOptions.setDelayedWriteRate()` for more information.
+The default value is 16 MB.
+|`nifi.flowfile.repository.rocksdb.max.background.flushes`
+|Specifies the maximum number of concurrent background flush jobs. See
RocksDB `DBOptions.setMaxBackgroundFlushes()` / `max_background_flushes` for
more information.
+The default value is 1.
+|`nifi.flowfile.repository.rocksdb.max.background.compactions`
+|Specifies the maximum number of concurrent background compaction jobs. See
RocksDB `DBOptions.setMaxBackgroundCompactions()` /
`max_background_compactions` for more information.
+The default value is 1.
+|`nifi.flowfile.repository.rocksdb.min.write.buffer.number.to.merge`
+|The minimum number of write buffers to merge together before writing to
storage. See RocksDB `ColumnFamilyOptions.setMinWriteBufferNumberToMerge()` /
`min_write_buffer_number_to_merge` for more information.
+The default value is 1.
+|`nifi.flowfile.repository.rocksdb.stat.dump.period`
+|The period at which to dump rocksdb.stats to the log. See RocksDB
`DBOptions.setStatsDumpPeriodSec()` / `stats_dump_period_sec` for more
information.
+The default value is 600 sec.
+|====
+
=== Swap Management
NiFi keeps FlowFile information in memory (the JVM)
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java
index 779c29c..1337511 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java
@@ -515,7 +515,7 @@ public class WriteAheadFlowFileRepository implements
FlowFileRepository, SyncLis
updateRepository(repoRecords, true);
synchronized (this.swapLocationSuffixes) {
-
this.swapLocationSuffixes.remove(normalizeSwapLocation(swapLocation));
+ this.swapLocationSuffixes.add(normalizeSwapLocation(swapLocation));
}
logger.info("Repository updated to reflect that {} FlowFiles were
swapped in to {}", new Object[]{swapRecords.size(), queue});
