Mark, your answer definitely helps. If I understand it correctly, you suggest that we change the default logging level, and the level of "org.apache.nifi.processors" to WARN, so that we will get a leaner log.
On the data provenance, I really like what NiFi offers. The question is whether there is a way to see that information outside of NiFi. The point is that the following settings limit the size of data provenance: nifi.provenance.repository.max.storage.time=24 hours nifi.provenance.repository.max.storage.size=1 GB Huagen On Fri, Jun 10, 2016 at 12:53 PM, Mark Payne <[email protected]> wrote: > Hi Huagen, > > This is typically the type of logging you will see in NiFi. Each processor > will generally log at an INFO > level what it is doing for each FlowFile. Unfortunately, though, this can > become extremely verbose, > and many people want that logging toned down, so in the master branch of > NiFi, the minimum log > level for processors is set to WARN instead of INFO. You can change this > by updating the conf/logback.xml > and setting the log level of "org.apache.nifi.processors" to INFO instead > of WARN. > > The main reason that we have changed the default log level though is that > in NiFi, it is very rare to need > to go through all of the tedious labor of grepping through logs. Instead, > the recommended approach is to > use the Data Provenance features [1]. This will allow you to search for > data of interest to you and see exactly > how it was processed throughout the flow. Additionally, this provides you > access to the FlowFile attributes as > they were each step along the way, and the ability to click-to-content to > see how the data looked at that point > in the flow as well. It also allows you to visualize what happened to the > data, even if it is split into many smaller > pieces of data (potentially with different filenames) or merged together > with other data, so that you don't have > to worry about the filename. > > I hope this helps! > > -Mark > > [1] > http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance > > > On Jun 10, 2016, at 11:58 AM, Huagen peng <[email protected]> wrote: > > Hi, > > I would like to learn about some better practices on logging. Here is > what I would imagine in an ideal log for a flow like fetching files from > SFTP, processing the files in certain way, and then saving the file to the > disk. In the log, I would see that the SFTP step is triggered, with the > filename in clear text. I would then see that the file processing is > started, and that the file is saved. If there are errors, I would also see > the errors in the log as well. > > How would I achieve that or something close to that in NiFi? > > Thanks, > > Huagen > > >
