Hi guys, I’m jumping in here, we see the same as Giulia, just in another timeframe. We are using the default provenance settings (nothing changed except the path) with 4GB space on the provenance partition. Are there any suggestions what to change? Of course as I’m checking it now we should lift up the two values below to fit better to the partition size of 4GB.
nifi.provenance.repository.max.storage.size=1 GB nifi.provenance.repository.index.shard.size =500 MB What else? [user@abc ~]$ du -h /provenance_repo/ 144K /provenance_repo/toc 273M /provenance_repo/index-1564098506268 400M /provenance_repo/ Thanks in advance, Josef From: Mark Payne <[email protected]> Reply to: "[email protected]" <[email protected]> Date: Monday, 19 August 2019 at 15:49 To: "[email protected]" <[email protected]> Subject: Re: "Deadlock" data provenance after few days Giulia, Looking at the thread dump, it appears that there are no problems with deadlocking or anything like that in Provenance. As Joe mentioned, the settings may need to be adjusted a bit. Can you share what you have in nifi.properties in the Provenance section? Also, it would be helpful to provide the results of running the following command on your provenance repo directory: du -h provenance_repository/ The default settings that are used for provenance are intended to avoid using a huge amount of disk space. However, they are not ideal for production. The max storage size is set to 1 GB with the shard size set to 500 MB. As a result, depending on timing, it may be possible that the provenance index directories occupy all - or nearly all - of the 1 GB storage size allocated and as a result, all of the event data is aged off. If this happens, you'll see everything functioning properly except that provenance events won't be shown. Thanks -Mark On Aug 19, 2019, at 7:59 AM, Joe Witt <[email protected]<mailto:[email protected]>> wrote: Just to clarify when you say locked fo you mean that you just cannot get any provenance results but the flow still functions? If so I think your prov settings need to be adjusted. Ill review the thread output when not on a phone. Thanks On Mon, Aug 19, 2019 at 4:24 AM Giulia Scalaberni <[email protected]<mailto:[email protected]>> wrote: Hi all, As requested by Joe Witt, here I attach the dump log after the lock. My Nifi after nearly 30 days of scheduled activity has locked as usual… BG, Giulia Da: Joe Witt [mailto:[email protected]<mailto:[email protected]>] Inviato: giovedì 1 agosto 2019 15:31 A: [email protected]<mailto:[email protected]> Oggetto: Re: "Deadlock" data provenance after few days Giulia, When you're experiencing this condition can you capture a thread dump and share the logs (bootstrap/app)? To create this you can run <where-nifi-installed>/bin/nifi.sh dump threaddump-locked-prov.log Thanks On Thu, Aug 1, 2019 at 3:57 AM Giulia Scalaberni <[email protected]<mailto:[email protected]>> wrote: Hi, Both on my 1.9.2 and 1.9.0 standalone installations data provenance is working correctly for nearly a month scheduled once a day then on the server all the data provenance in all individual processors and in the general reporting stays empty, even when I send new data through all the processors manually. When opening a data provenance menu I see a very short popup menu appearing, but it closes for me too fast to see what is going on. On disk I still have more than 100GB available. The problem goes away when I restart the Nifi instance, and pops up again after nearly a month. Does anyone has the same behavior? Have you find a solution? Thank you in advance. Have a nice day, Giulia
smime.p7s
Description: S/MIME Cryptographic Signature
