Vinz2168 commented on PR #11912: URL: https://github.com/apache/nifi/pull/11912#issuecomment-6972376225
Thanks for the review @exceptionfactory, that's a fair point, so let me clarify the scope of the change and the use case. **What the change actually involves** The PR does not touch any MiNiFi code. It only: 1. adds the existing `nifi-per-process-group-logging` jar (~8 KB, a single `Discriminator` class depending only on slf4j/logback, which are already in `lib/`) to the MiNiFi assembly, and 2. adds a `SiftingAppender` definition to the default `logback.xml`. The framework side is already present in MiNiFi today: `StandardComponentLog` already resolves the Process Group log file suffix and attaches it to the logging event. The only missing piece is the discriminator class on the classpath. That's why the manual workaround needs both copying the jar into `lib/` and editing `logback.xml`, and the jar step has to be repeated on every upgrade. **Footprint / default behavior** I agree that per-PG logging shouldn't be a default for MiNiFi, so I've added a commit to the PR that makes it opt-in: - the jar is shipped in `lib/`; - the `DEDICATED_LOGGING` appender and its `appender-ref` are included in `logback.xml` but **commented out**, with a short note on how to enable them. With the block commented out, logback never instantiates the appender and the class is never loaded, so the runtime behavior and the processing overhead are exactly the same as today. Users who need the feature uncomment one block, and nothing has to be copied into the installation. For reference, with the appender enabled (logback 1.6.6, 1M INFO events to file) the overhead I measured is about 1–1.5%, dominated by file I/O. **Use case** The concrete need is traceability of the processing activity of a specific flow. The flow uses `LogMessage` (and component log messages in general) inside a Process Group to record what was processed, e.g. which files/records were handled and their outcome. That trail has to be kept for a longer period than the operational logs. Today the only option is to increase the retention of `minifi-app.log` (`maxHistory` / `totalSizeCap`). That means keeping a much larger amount of framework and component noise just to preserve a small number of relevant messages, which on an edge device directly translates into more disk usage. With per-PG logging, the messages of that Process Group are also written to a dedicated `minifi-app-<suffix>.log`, which has its own rolling policy. The dedicated file can then have a long retention while being small (it contains only that PG's messages), and `minifi-app.log` keeps its current, short retention. It also helps with troubleshooting on a running agent: when several Process Groups are active, isolating the messages of the one under investigation from `minifi-app.log` currently requires manual filtering, while a dedicated file gives that view directly. In that sense the feature actually helps keep the overall footprint smaller than the alternative. **Side note** While testing I found that `NifiDiscriminator` throws a `NullPointerException` for every event without key/value pairs, and logback swallows it. A plain null check is not a valid fix, because the `SiftingAppender` would then route all those events to a `...-logFileSuffix_IS_UNDEFINED.log` file. The proper fix is a small `Filter` that denies events without the suffix. Since it affects NiFi as well, I'll open a separate Jira for it rather than expanding the scope of this PR. Let me know if the opt-in approach works for you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
