You can already correlate logs per processor using the id, but that's a bit clunky especially with multi line messages (eg stack traces). It might be possible to add the processor group id to the logging context I guess.
Another use case is monitoring production flows for intermittent errors when the bulletins might be missed. Provenance tells us what happened, but the logs tell us why it happened (tho some processors populate extra attributes). All too often we find the failure connections are automatically terminated :-( I've been putting off integrating NiFi into our ELK stack, but maybe I ought to do that anyway. I'll try to write up a jira. On Fri, 7 Sep 2018, 20:14 Joe Witt, <[email protected]> wrote: > Yeah makes sense. We could either make the process group specific > logs an option, or make tag based logs an option, or we could perhaps > make bulletins something we store/make accessible for a longer period > of time. With the first two it is probably not a lot of code changes > but then raw logs the user has to correlate to specific components in > the flow. The bulletin path is probably a good bit more code though a > better user/operator experience. > > But your point/ask makes sense. > On Fri, Sep 7, 2018 at 3:04 PM James Srinivasan > <[email protected]> wrote: > > > > Sure, we have several data flow managers who are each responsible for a > number of separate data flows, each of which lives in a separate processor > group. When debugging, being able to see what's happening in your processor > group (including full stack traces) is essential, but the noise from all > the other processors is hugely distracting. > > > > On Fri, 7 Sep 2018, 19:52 Joe Witt, <[email protected]> wrote: > >> > >> James, > >> > >> One log per process group is not currently supported. Through things > >> like the message diagnostic context it might be pretty doable. > >> > >> That said, the logs aren't really meant to be the true historical > >> record of value. The data provenance entries are. Can you share more > >> about what you're trying to accomplish? > >> > >> Thanks > >> On Fri, Sep 7, 2018 at 2:47 PM James Srinivasan > >> <[email protected]> wrote: > >> > > >> > As we add more and more different processor groups doing separate > things, our nifi-app log is getting rather unmanageable. I'm probably > missing something obvious, but is one log per group possible? > >> > > >> > Thanks >
