szaszm commented on code in PR #1713:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1713#discussion_r1447296805
##########
CONFIGURE.md:
##########
@@ -560,3 +631,92 @@ To notify the agent which extensions it should load see
[Loading extensions](Ext
### JNI Functionality
Please see the [JNI Configuration Guide](JNI.md).
+
+## Log configuration
+By default the application logs for Apache MiNiFi C++ can be found in the
${MINIFI_HOME}/logs/minifi-app.log file with default INFO level logging. The
logger can be reconfigured in the ${MINIFI_HOME}/conf/minifi-log.properties
file to use different output streams, log level, and output format.
+
+### Log appenders
+In the configuration file it is possible to define different output streams by
defining what type of appenders (log sinks) should be used for logging. The
following appenders are supported:
+- rollingappender - logs written to this log sink are written to a file with a
rolling policy
+- stdout - logs written to this log sink are written to the standard output
+- stderr - logs written to this log sink are written to the standard error
+- syslog - logs written to this log sink are written to the syslog
+- alert - logs written to this log sink are critical logs forwarded through
HTTP as alerts
+- nullappender - logs written to this log sink are discarded
+
+Defining a new appender can be done in the format `appender.<appender
name>=<appender type>` followed by defining its properties. For example here is
how to define a rolling file appender:
+
+ # in minifi-log.properties
+ appender.rolling=rollingappender
+ appender.rolling.directory=${MINIFI_HOME}/logs
+ appender.rolling.file_name=minifi-app.log
+ appender.rolling.max_files=3
+ appender.rolling.max_file_size=5 MB
+
+Here's an example of an alert appender with its available properties:
+
+ # in minifi-log.properties
+ appender.alert1=alert
+ appender.alert1.url=<URL>
+ appender.alert1.filter=<regex pattern to match logs against>
+ appender.alert1.rate.limit=10 min
+ appender.alert1.flush.period=5 s
+ appender.alert1.batch.size=100 KB
+ appender.alert1.buffer.limit=1 MB
+ appender.alert1.level=TRACE
+ appender.alert1.ssl.context.service=<Name of the SSLContextService>
+
+### Log levels
+After the appenders are defined you can set the log level and logging target
for each of them. Appenders can be set to log everything on a specific log
level, but you can also define some appenders to only write logs from specific
namespaces or classes with different log levels. The log level can be set to
one of the following values: OFF, TRACE, DEBUG, INFO, WARN, ERROR. The log
levels can be set in the following way:
+
+ # in minifi-log.properties
+ # Set the rolling appender to log everything with INFO level
+ logger.root=INFO,rolling
+ # But only write logs from the org::apache::nifi::minifi::FlowController
class with ERROR level
+
logger.org::apache::nifi::minifi::core::logging::LoggerConfiguration=ERROR,rolling
Review Comment:
That's a different class
##########
CONFIGURE.md:
##########
@@ -287,6 +341,60 @@ Another option to define a state storage is to use the
following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
+- The Flow File Repository holds the metadata (like flow file attributes) of
the flow Files.
+- The Content Repository holds the content of the flow files.
+- The Provenance Repository holds the history of the flow files.
+
+The underlying implementation to use for these repositories can be configured
in the minifi.properties file.
+
+The Flow File Repository can be configured with the
`nifi.flowfile.repository.class.name` property. If not specified, it uses the
`FlowFileRepository` class by default, which stores the flow file metadata in a
RocksDB database. Alternatively it can be configured to use a
`VolatileFlowFileRepository` that keeps the state in memory and flushes upon
restart, or the `NoOpRepository` for not keeping any state.
+
+ # in minifi.properties
+ nifi.flowfile.repository.class.name=VolatileFlowFileRepository
+
+The Content Repository can be configured with the
`nifi.content.repository.class.name` property. If not specified, it uses the
`DatabaseContentRepository` class by default, which persists the content in a
RocksDB database. `DatabaseContentRepository` is also the default value
specified in the minifi.properties file. Alternatively it can be configured to
use a `VolatileContentRepository` that keeps the state in memory and flushes
upon restart, or the `FileSystemRepository` to keep the state in regular files.
+
+**NOTE:** RocksDB database has a limit of 4GB for the size of a database
object. Due to this if you are expecting to process larger flow files than 4GB
you should use the `FileSystemRepository`. The downside of using
`FileSystemRepository` is that it does not have the transactional guarantees of
the RocksDB repository implementation.
+
+ # in minifi.properties
+ nifi.content.repository.class.name=FileSystemRepository
+
+The Provenance Repository can be configured with the
`nifi.provenance.repository.class.name` property. If not specified, it uses the
`ProvenanceRepository` class by default, which persists the provenance events
in a RocksDB database. Alternatively it can be configured to use a
`VolatileProvenanceRepository` that keeps the state in memory and flushes upon
restart, or the `NoOpRepository` to not keep track of the provenance events. By
default we do not keep track of the provenance data, so `NoOpRepository` is the
value specified in the default minifi.properties file.
+
+ # in minifi.properties
+ nifi.flowfile.repository.class.name=NoOpRepository
+
+
+### Configuring Volatile Repositories
+As stated before each of the repositories can be configured to be volatile
(state kept in memory and flushed upon restart) or persistent. Volatile
repositories have some additional options, that can be specified in the
following ways:
Review Comment:
If I remember correctly, volatile repos can behave in unexpected ways, when
reaching their limits, probably breaking the configuration. Maybe
@adamdebreceni remembers how exactly they break, from #1435 ?
If so, we should probably document this behavior.
##########
README.md:
##########
@@ -154,7 +154,7 @@ and rebuild.
* perl -- Required for OpenSSL configuration
* NASM -- Required for OpenSSL only on Windows
-**NOTE** On Windows if Strawberry Perl is used the
`${StrawberryPerlRoot}\c\bin` directory should not be part of the %PATH%
variable or make sure to find git's patch executable before that as Strawberry
Perl's patch.exe will be found as the patch executable in the configure phase
instead of the git patch executable. Also on Windows CMake's CPack is used for
MSI generation, building WIX files and calling WIX toolset tools to create an
MSI. If Chocolatey package manager is used its CPack can conflict with CMake,
so make sure that CMake's CPack is found in the %PATH% before that.
+**NOTE:** On Windows if Strawberry Perl is used the
`${StrawberryPerlRoot}\c\bin` directory should not be part of the PATH
environment variable. Alternatively make sure that git's patch executable path
comes before Strawberry Perl's patch.exe path in the PATH environment variable,
so that the git patch executable is found in the in the CMake configuration
phase. Also on Windows CMake's CPack is used for MSI generation, building WIX
files and calling WIX toolset tools to create an MSI. If Chocolatey package
manager is used its CPack can conflict with CMake, so make sure that CMake's
CPack is found in the %PATH% before that.
Review Comment:
I found it not to be enough to move the Strawberry Perl c/bin in the path,
I had to actually rename or remove the patch.exe on my VM to not cause any
issues.
I think we should recommend the scoop package of perl: it's strawberry perl
internally, but doesn't pollute the path with its toolchain, and doesn't cause
these issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
