Thanks, Uwe. What I see in the provenance results is that you have 1,000 of the maximum 1,000 results shown. This suggests that there are more than 1,000 records available for the current search criteria. When you sort by the Date/Time column, this sorts only through the 1,000 visible records, not the underlying provenance database.
Can you change the search criteria to focus on a particular time window, perhaps 1 hour of data in the missing period? It should return less than 1,000 records, but hopefully more than 0. Thanks, James On Thu, Mar 30, 2017 at 12:06 AM, Uwe Geercken <[email protected]> wrote: > James, > > here is the screenshot of the data provenance: > > http://imgur.com/a/29M8I > > and here my flow: > > http://imgur.com/a/nd6v7 > > > If I clear the search in provenance it is still not showing events from > the TailFile processor although it is running. > > Rgds, > > Uwe > > *Gesendet:* Freitag, 24. März 2017 um 19:27 Uhr > *Von:* "James Wing" <[email protected]> > *An:* [email protected] > *Betreff:* Re: Nifi - Data provenance not reporting anymore > Uwe, > > Can you share a screenshot of the provenance search criteria? If you try > the search the other way around, starting with all provenance events and > then filtering down to the time and processor, does that change anything? > > Thanks, > > James > > On Fri, Mar 24, 2017 at 8:39 AM, Uwe Geercken <[email protected]> > wrote: >> >> Hello, >> >> I have a production flow in place (nifi-1.1.1-RC1). It runs on a virtual >> machine running redhat. The reads multiple Json files (small) from a >> windows share/mountpoint. Various shell scripts write single lines (json) >> into files. The files are read, split and put into MongoDb. >> >> The flow goes like this: >> >> TailFile >> Split Text >> PutMongo >> >> Today is 2017-03-24. >> >> I have a webpage running which displays the data from the MongoDb. So I >> know when data arrives. And I can see that data arrived throughout the last >> days and also today >> >> I have detected something I don't understand: TailFile shows me in the >> flow that it has processed flow file in the last 5 minutes. But when I show >> the provenance events then Nifi shows: "Oldest event available: 03/16/2017 >> 18:02:34 CET". The list of events below it shows the most recent event as >> "03/24/2017 11:00:35.290 CET". But where are the events since then? I see >> that I have data in MongoDb after that. >> >> My current nifi-app.log looks like this: >> >> 2017-03-24 16:28:38,472 INFO [Write-Ahead Local State Provider >> Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali. >> MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 >> Swap Files in 27 milliseconds (Stop-the-world time = 1 milliseconds, Clear >> Edit Logs time = 12 millis), max Transaction ID 12245824 >> 2017-03-24 16:30:38,506 INFO [Write-Ahead Local State Provider >> Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali. >> MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 >> Swap Files in 34 milliseconds (Stop-the-world time = 2 milliseconds, Clear >> Edit Logs time = 15 millis), max Transaction ID 12247564 >> 2017-03-24 16:32:38,540 INFO [Write-Ahead Local State Provider >> Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali. >> MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 >> Swap Files in 33 milliseconds (Stop-the-world time = 1 milliseconds, Clear >> Edit Logs time = 11 millis), max Transaction ID 12248871 >> >> There are no errors in the log and when I look back around "03/24/2017 >> 11:00:35.290 CET" there is nothing special either: >> >> 2017-03-24 10:56:32,547 INFO [Write-Ahead Local State Provider >> Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali. >> MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 >> Swap Files in 34 milliseconds (Stop-the-world time = 2 milliseconds, Clear >> Edit Logs time = 17 millis), max Transaction ID 12002746 >> 2017-03-24 10:58:32,580 INFO [Write-Ahead Local State Provider >> Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali. >> MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 >> Swap Files in 33 milliseconds (Stop-the-world time = 2 milliseconds, Clear >> Edit Logs time = 16 millis), max Transaction ID 12004378 >> 2017-03-24 11:00:32,624 INFO [Write-Ahead Local State Provider >> Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali. >> MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 >> Swap Files in 43 milliseconds (Stop-the-world time = 2 milliseconds, Clear >> Edit Logs time = 20 millis), max Transaction ID 12005794 >> 2017-03-24 11:00:35,339 INFO [Timer-Driven Process Thread-7] >> o.a.n.p.PersistentProvenanceRepository >> Created new Provenance Event Writers for events starting with ID 2237386 >> 2017-03-24 11:00:50,975 INFO [Provenance Repository Rollover Thread-1] >> o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal >> files (111869 records) into single Provenance Log File >> ./provenance_repository/2125517.prov in 15677 milliseconds >> 2017-03-24 11:00:50,976 INFO [Provenance Repository Rollover Thread-1] >> o.a.n.p.PersistentProvenanceRepository Successfully Rolled over >> Provenance Event file containing 185608 records. In the past 5 minutes, 50 >> events have been written to the Provenance Repository, totaling 37.05 KB >> 2017-03-24 11:02:32,658 INFO [Write-Ahead Local State Provider >> Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali. >> MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 >> Swap Files in 33 milliseconds (Stop-the-world time = 1 milliseconds, Clear >> Edit Logs time = 16 millis), max Transaction ID 12007317 >> 2017-03-24 11:04:32,687 INFO [Write-Ahead Local State Provider >> Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali. >> MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 >> Swap Files in 28 milliseconds (Stop-the-world time = 1 milliseconds, Clear >> Edit Logs time = 14 millis), max Transaction ID 12008733 >> >> >> So the flow is working but I don't have provenance events. Can somebody >> help? >> >> Rgds, >> >> Uwe >> >> >
