Vincenzo Lombardo created NIFI-16602:
----------------------------------------
Summary: NifiDiscriminator throws NullPointerException for every
log event without key/value pairs
Key: NIFI-16602
URL: https://issues.apache.org/jira/browse/NIFI-16602
Project: Apache NiFi
Issue Type: Bug
Reporter: Vincenzo Lombardo
h3. Description
{{org.apache.nifi.logging.NifiDiscriminator}} (module
{{nifi-per-process-group-logging}}) iterates over
{{ILoggingEvent.getKeyValuePairs()}} without checking for {{null}}. Logback
returns {{null}} for events that carry no key/value pairs, which covers every
framework log message and every component log message from Process Groups
without a Log File Suffix. So for the vast majority of events the discriminator
throws a {{NullPointerException}}:
{noformat}
ERROR in ch.qos.logback.classic.sift.SiftingAppender[DEDICATED_LOGGING] -
Appender [DEDICATED_LOGGING] failed to append.
java.lang.NullPointerException: Cannot invoke "java.util.List.iterator()"
because the return value of
"ch.qos.logback.classic.spi.ILoggingEvent.getKeyValuePairs()" is null
at
org.apache.nifi.logging.NifiDiscriminator.getDiscriminatingValue(NifiDiscriminator.java:30)
at
ch.qos.logback.core.sift.SiftingAppenderBase.append(SiftingAppenderBase.java:113)
at ch.qos.logback.core.AppenderBase.doAppend(AppenderBase.java:85)
{noformat}
The exception is caught by {{AppenderBase.doAppend}}, so the event is not
written to the dedicated file and the visible output is correct. However:
* an exception is created and thrown for every log event that goes through the
{{DEDICATED_LOGGING}} appender, which is enabled by default in the NiFi
{{logback.xml}};
* "failed to append" errors are recorded in the logback status;
* the correct output relies on the exception rather than on explicit logic.
h3. Why a null check in the discriminator is not a valid fix
If {{getDiscriminatingValue}} returns {{null}}, {{SiftingAppenderBase.append}}
does not skip the event. It routes the event to an appender created for the
undefined key. All events without a suffix then end up in an additional
{{nifi-app-logFileSuffix_IS_UNDEFINED.log}} file, which duplicates the whole
application log and doubles the write I/O.
h3. Proposed fix
Add a small {{Filter<ILoggingEvent>}} to {{nifi-per-process-group-logging}}
that returns {{DENY}} for events without the {{logFileSuffix}} key/value pair
and {{NEUTRAL}} otherwise. Then reference it from the {{DEDICATED_LOGGING}}
appender:
{code:xml}
<appender name="DEDICATED_LOGGING"
class="ch.qos.logback.classic.sift.SiftingAppender">
<filter class="org.apache.nifi.logging.LogFileSuffixFilter"/>
<discriminator class="org.apache.nifi.logging.NifiDiscriminator"/>
...
</appender>
{code}
Filters run in {{AppenderBase.doAppend}} before {{append}}, so events without a
suffix never reach the discriminator.
Affected configuration files:
*
{{nifi-framework-bundle/.../nifi-resources/src/main/resources/conf/logback.xml}}
* MiNiFi {{minifi-resources/src/main/resources/conf/logback.xml}} (the block is
commented out, see NIFI-16572)
The discriminator should keep its current semantics. Existing installations
often keep their {{logback.xml}} across upgrades and therefore won't have the
new filter, and making the discriminator return {{null}} would produce the
{{IS_UNDEFINED}} file described above.
h3. Measurements
Standalone test with logback 1.6.6 and slf4j 2.0.20: 1M INFO events to a file
appender, plus a {{SiftingAppender}} using {{NifiDiscriminator}}. Average of 3
runs:
||Configuration||Time||Files||
|No dedicated appender|~2.62 s|app.log|
|Dedicated appender, current discriminator (NPE path)|~2.66 s|app.log,
app-<suffix>.log|
|Dedicated appender, null check in discriminator|~5.2 s|app.log,
app-<suffix>.log, app-logFileSuffix_IS_UNDEFINED.log (all events)|
|Dedicated appender, proposed filter|~2.65 s|app.log, app-<suffix>.log|
On a modern JIT the performance difference between the current behavior and the
filter is within noise, because hot implicit NPEs are optimized. The main
benefit is correctness and clean logback status, not throughput.
h3. Related
NIFI-16572
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)