Vinz2168 commented on PR #11912:
URL: https://github.com/apache/nifi/pull/11912#issuecomment-6972376225

   Thanks for the review @exceptionfactory, that's a fair point, so let me 
clarify the scope of the change and the use case.
   
   **What the change actually involves**
   
   The PR does not touch any MiNiFi code. It only:
   1. adds the existing `nifi-per-process-group-logging` jar (~8 KB, a single 
`Discriminator` class depending only on slf4j/logback, which are already in 
`lib/`) to the MiNiFi assembly, and
   2. adds a `SiftingAppender` definition to the default `logback.xml`.
   
   The framework side is already present in MiNiFi today: 
`StandardComponentLog` already resolves the Process Group log file suffix and 
attaches it to the logging event. The only missing piece is the discriminator 
class on the classpath. That's why the manual workaround needs both copying the 
jar into `lib/` and editing `logback.xml`, and the jar step has to be repeated 
on every upgrade.
   
   **Footprint / default behavior**
   
   I agree that per-PG logging shouldn't be a default for MiNiFi, so I've added 
a commit to the PR that makes it opt-in:
   - the jar is shipped in `lib/`;
   - the `DEDICATED_LOGGING` appender and its `appender-ref` are included in 
`logback.xml` but **commented out**, with a short note on how to enable them.
   
   With the block commented out, logback never instantiates the appender and 
the class is never loaded, so the runtime behavior and the processing overhead 
are exactly the same as today. Users who need the feature uncomment one block, 
and nothing has to be copied into the installation.
   
   For reference, with the appender enabled (logback 1.6.6, 1M INFO events to 
file) the overhead I measured is about 1–1.5%, dominated by file I/O.
   
   **Use case**
   
   The concrete need is traceability of the processing activity of a specific 
flow. The flow uses `LogMessage` (and component log messages in general) inside 
a Process Group to record what was processed, e.g. which files/records were 
handled and their outcome. That trail has to be kept for a longer period than 
the operational logs.
   
   Today the only option is to increase the retention of `minifi-app.log` 
(`maxHistory` / `totalSizeCap`). That means keeping a much larger amount of 
framework and component noise just to preserve a small number of relevant 
messages, which on an edge device directly translates into more disk usage.
   
   With per-PG logging, the messages of that Process Group are also written to 
a dedicated `minifi-app-<suffix>.log`, which has its own rolling policy. The 
dedicated file can then have a long retention while being small (it contains 
only that PG's messages), and `minifi-app.log` keeps its current, short 
retention.
   
   It also helps with troubleshooting on a running agent: when several Process 
Groups are active, isolating the messages of the one under investigation from 
`minifi-app.log` currently requires manual filtering, while a dedicated file 
gives that view directly.
   
   In that sense the feature actually helps keep the overall footprint smaller 
than the alternative.
   
   **Side note**
   
   While testing I found that `NifiDiscriminator` throws a 
`NullPointerException` for every event without key/value pairs, and logback 
swallows it. A plain null check is not a valid fix, because the 
`SiftingAppender` would then route all those events to a 
`...-logFileSuffix_IS_UNDEFINED.log` file. The proper fix is a small `Filter` 
that denies events without the suffix. Since it affects NiFi as well, I'll open 
a separate Jira for it rather than expanding the scope of this PR.
   
   Let me know if the opt-in approach works for you.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to