Yes, each node has its persistent stores for each of those directories. On Sat, Apr 11, 2020 at 10:20 AM Patrick Timmins <[email protected]> wrote:
> Is the underlying storage for the four repositories (provenance, > database, flowfile, and content) consistent within a node? > > Are all three nodes in the cluster using the same type of underlying > storage/device for the various NiFi repositories? > > > On 4/11/2020 8:45 AM, Wyllys Ingersoll wrote: > > Nope, already checked that. > > On Fri, Apr 10, 2020 at 8:23 PM Patrick Timmins <[email protected]> wrote: > >> No issues here. Sounds like a timezone / system clock / clock drift >> issue (in a cluster). >> On 4/10/2020 11:59 AM, Joe Witt wrote: >> >> The provenance repo is in large scale use by many many users so >> fundamentally it does work. There are conditions that apparently need >> improving. In the past couple days these items have been flagged by folks >> on this list, JIRAs and PRs raised and merged, all good. If you can help by >> creating a build of the latest and confirm it fixes your case then please >> do so. >> >> Thanks >> >> On Fri, Apr 10, 2020 at 12:48 PM Darren Govoni <[email protected]> >> wrote: >> >>> It would seem the feature is either broken completely or only works in >>> specific conditions. >>> >>> Can the Nifi team put a fix on their road map for this? >>> Its a rather central feature to Nifi. >>> >>> Sent from my Verizon, Samsung Galaxy smartphone >>> >>> ------------------------------ >>> *From:* Wyllys Ingersoll <[email protected]> >>> *Sent:* Friday, April 10, 2020 11:17:42 AM >>> *To:* [email protected] <[email protected]> >>> *Subject:* Re: Not Seeing Provenance data >>> >>> I have a similar problem with viewing provenance. I have a 3-node >>> cluster in a kubernetes environment, the provenance_repository directory >>> for each node is on a persistent data store so it is not deleted or lost >>> between container restarts (which are not very common). My >>> nifi.provenance.repository.max.storage.time is 24 hours. >>> >>> Whenever I try to view any provenance, nothing is ever shown. If I >>> manually inspect the provenance_repository directory, there is a lucene >>> index and TOC being created. >>> >>> I see log messages like these: >>> >>> Submitting query +processorId:882133fe-b684-148b-ad88-7850437ca591 with >>> identifier 64a703fe-0171-1000-0000-000065abd91a against index directories >>> [./provenance_repository/lucene-8-index-1560864819888] >>> Returning the following list of index locations because they were >>> finished being written to before 1586531601311: [] >>> Found no events in the Provenance Repository. In order to perform >>> maintenace of the indices, will assume that the first event time is now >>> (1586531601311) >>> >>> >>> Any suggestions? >>> >>> -Wyllys Ingersoll >>> >>> >>> >>> On Thu, Apr 9, 2020 at 11:25 AM Dobbernack, Harald (Key-Work) < >>> [email protected]> wrote: >>> >>> Hey Mark, >>> >>> >>> >>> great news and thank you very much! >>> >>> >>> >>> Happy Holidays! >>> >>> Harald >>> >>> >>> >>> *Von:* Mark Payne <[email protected]> >>> *Gesendet:* Donnerstag, 9. April 2020 17:18 >>> *An:* [email protected] >>> *Betreff:* Re: Not Seeing Provenance data >>> >>> >>> >>> Thanks Harald, >>> >>> >>> >>> I have created a Jira [1] for this. There’s currently a PR up for it as >>> well. >>> >>> >>> >>> Thanks >>> >>> -Mark >>> >>> >>> >>> [1] https://issues.apache.org/jira/browse/NIFI-7346 >>> >>> >>> >>> On Apr 9, 2020, at 11:14 AM, Dobbernack, Harald (Key-Work) < >>> [email protected]> wrote: >>> >>> >>> >>> Hi Mark, >>> >>> >>> >>> I can confirm after testing that if no provenance event has been >>> generated in a time greater than the set >>> nifi.provenance.repository.max.storage.time >>> then as expected the last recorded provenance events don’t exist anymore >>> but also from then on any new provenance events are also not searchable, >>> the provenance Search remains completely empty regardless of how many flows >>> are active. As described also *.prov file is then missing in provenance >>> repository. After restart of Nifi new prov File will be generated and >>> provenance will work again, but only showing stuff generated since last >>> NiFi Start. >>> >>> >>> >>> So yes, I’d say your Idea >>> >>> ‘If so, then I think that would understand why it deleted the data. >>> It’s trying to age off old data >>> >>> but unfortunately it doesn’t perform a check to first determine >>> whether or not the “old file” >>> >>> that it’s about to delete is also the “active file”.’ >>> >>> fits very nicely to my test. >>> >>> >>> >>> As a workaround we’re going to set a greater >>> nifi.provenance.repository.max.storage.time >>> until this can be resolved. >>> >>> >>> >>> Thanks again for looking into this. >>> >>> Harald >>> >>> >>> >>> >>> >>> *Von:* Dobbernack, Harald (Key-Work) >>> *Gesendet:* Donnerstag, 9. April 2020 15:22 >>> *An:* [email protected] >>> *Betreff:* AW: Not Seeing Provenance data >>> >>> >>> >>> Hi Mark, >>> >>> >>> >>> thank you for looking into this. >>> >>> >>> >>> The nifi.provenance.repository.max.storage.time setting might explain >>> why I haven’t been experiencing the effect so often since changing from the >>> default to 120 hours a few months ago 😉 >>> >>> >>> >>> But I believe provenance stopped working last time although there was an >>> ‘active’ flows in wait Processor, expiring every hour, going on to ‘send a >>> message’ before being rerouted to the same wait processor. I would have >>> expected this generates provenance entries? As I am not actually 100% sure >>> if that wait processor was in use when last provenance got lost I will >>> check with a testing system to see if I can reproduce provenance breakage >>> when no active flows are around for a time greater >>> nifi.provenance.repository.max.storage.time and I will get back to you. >>> >>> >>> >>> Thank you! >>> >>> Harald >>> >>> >>> >>> >>> >>> *Von:* Mark Payne <[email protected]> >>> *Gesendet:* Donnerstag, 9. April 2020 14:41 >>> *An:* [email protected] >>> *Betreff:* Re: Not Seeing Provenance data >>> >>> >>> >>> Hey Daren, Herald, >>> >>> >>> >>> Thanks for the note. I have seen this once before but couldn’t figure >>> out what caused it. Restarting addressed the issue. >>> >>> >>> >>> I think I may understand the problem, now, though, after looking at it >>> again. >>> >>> >>> >>> In nifi.properties, there are a couple of property named >>> “nifi.provenance.repository.max.storage.time” that defaults to “24 hours" >>> >>> Is it possible that you went 24 hours (or whatever value is set for that >>> property) without generating any Provenance events? >>> >>> >>> >>> If so, then I think that would understand why it deleted the data. It’s >>> trying to age off old data but unfortunately it doesn’t perform a check to >>> first determine whether or not the “old file” that it’s about to delete is >>> also the “active file”. >>> >>> >>> >>> Can you confirm whether or not you would expect to see 24 hours pass >>> without any provenance data? >>> >>> >>> >>> Thanks >>> >>> -Mark >>> >>> >>> >>> >>> >>> >>> >>> On Apr 9, 2020, at 4:32 AM, Dobbernack, Harald (Key-Work) < >>> [email protected]> wrote: >>> >>> >>> >>> What I noticed is that as long as provenance is working there will be >>> *.prov files in the directory. When Provenance isn’t working these files >>> are not to be seen. Maybe some Cleaning Process deletes those files >>> prematurely or the process building them doesn’t work any more? >>> >>> >>> >>> *Von:* Dobbernack, Harald (Key-Work) <[email protected]> >>> *Gesendet:* Donnerstag, 9. April 2020 10:27 >>> *An:* [email protected] >>> *Betreff:* AW: Not Seeing Provenance data >>> >>> >>> >>> This is something I experience too from time to time. My quick and dirty >>> workaround is stop nifi, delete everything in the provenance directory, >>> restart…. Then Provenance is usable again (of course only with data since >>> the delete) . I’m hoping very much there is a better way, someone can show >>> us better settings or a potential bug can be discovered… >>> >>> >>> >>> *Von:* Darren Govoni <[email protected]> >>> *Gesendet:* Mittwoch, 8. April 2020 20:31 >>> *An:* [email protected] >>> *Betreff:* Not Seeing Provenance data >>> >>> >>> >>> Hi, >>> >>> When I go to "View data provenance" in Nifi, I never see any logs for >>> my flow. Am I missing some configuration setting somewhere? >>> >>> >>> >>> thanks, >>> >>> Darren >>> >>> >>> >>> >>> >>> *Harald Dobbernack* >>> Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | >>> https://www.key-work.de | Datenschutz >>> <https://www.key-work.de/de/footer/datenschutz.html> >>> Fon: +49-721-78203-264 | E-Mail: [email protected] | Fax: >>> +49-721-78203-10 >>> >>> Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim >>> Geschäftsführer: Andreas Stappert, Tobin Wotring >>> >>> >>> >>>
