Giulia, Looking at the thread dump, it appears that there are no problems with deadlocking or anything like that in Provenance. As Joe mentioned, the settings may need to be adjusted a bit. Can you share what you have in nifi.properties in the Provenance section? Also, it would be helpful to provide the results of running the following command on your provenance repo directory:
du -h provenance_repository/ The default settings that are used for provenance are intended to avoid using a huge amount of disk space. However, they are not ideal for production. The max storage size is set to 1 GB with the shard size set to 500 MB. As a result, depending on timing, it may be possible that the provenance index directories occupy all - or nearly all - of the 1 GB storage size allocated and as a result, all of the event data is aged off. If this happens, you'll see everything functioning properly except that provenance events won't be shown. Thanks -Mark On Aug 19, 2019, at 7:59 AM, Joe Witt <[email protected]<mailto:[email protected]>> wrote: Just to clarify when you say locked fo you mean that you just cannot get any provenance results but the flow still functions? If so I think your prov settings need to be adjusted. Ill review the thread output when not on a phone. Thanks On Mon, Aug 19, 2019 at 4:24 AM Giulia Scalaberni <[email protected]<mailto:[email protected]>> wrote: Hi all, As requested by Joe Witt, here I attach the dump log after the lock. My Nifi after nearly 30 days of scheduled activity has locked as usual… BG, Giulia Da: Joe Witt [mailto:[email protected]<mailto:[email protected]>] Inviato: giovedì 1 agosto 2019 15:31 A: [email protected]<mailto:[email protected]> Oggetto: Re: "Deadlock" data provenance after few days Giulia, When you're experiencing this condition can you capture a thread dump and share the logs (bootstrap/app)? To create this you can run <where-nifi-installed>/bin/nifi.sh dump threaddump-locked-prov.log Thanks On Thu, Aug 1, 2019 at 3:57 AM Giulia Scalaberni <[email protected]<mailto:[email protected]>> wrote: Hi, Both on my 1.9.2 and 1.9.0 standalone installations data provenance is working correctly for nearly a month scheduled once a day then on the server all the data provenance in all individual processors and in the general reporting stays empty, even when I send new data through all the processors manually. When opening a data provenance menu I see a very short popup menu appearing, but it closes for me too fast to see what is going on. On disk I still have more than 100GB available. The problem goes away when I restart the Nifi instance, and pops up again after nearly a month. Does anyone has the same behavior? Have you find a solution? Thank you in advance. Have a nice day, Giulia
