Darren - I'm using NiFi 1.11.4 for a while now (with OOTB configuration) and I do have provenance data for the flows I'm running.
Le lun. 10 août 2020 à 23:24, Wyll Ingersoll < [email protected]> a écrit : > Ah! That fixed my problem! > > I am running a secure/authenticated configuration. My user had the > correct policy permissions when viewed from the hamburger menu, but not on > the access-policies for the individual processors/canvas. > > Very confusing! One would think if the admin user had the permissions, > they would apply everywhere, but apparently not. > > Anyway, now I can see provenance data. Thanks for the tip! > ------------------------------ > *From:* Shawn Weeks <[email protected]> > *Sent:* Monday, August 10, 2020 5:19 PM > *To:* [email protected] <[email protected]> > *Subject:* Re: Data Provenance Stops Working > > > That sounds like a permission issue. Are you running in secure mode? If so > right click on the main canvas and go to access policies and make sure your > current user is allowed provenance access, you’ll also have to go to the > policies section in the system menu in the upper right hand corner and do > the same. > > > > Thanks > > Shawn > > > > *From: *Darren Govoni <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Monday, August 10, 2020 at 3:52 PM > *To: *"[email protected]" <[email protected]> > *Subject: *Re: Data Provenance Stops Working > > > > I also use 1.11.4 and out of the box there IS NO provenance data > whatsoever. It just doesn't work if you install and run nifi as is. > > Sent from my Verizon, Samsung Galaxy smartphone > > Get Outlook for Android <https://aka.ms/ghei36> > > > ------------------------------ > > *From:* Shawn Weeks <[email protected]> > *Sent:* Monday, August 10, 2020 2:23:19 PM > *To:* [email protected] <[email protected]> > *Subject:* Re: Data Provenance Stops Working > > > > It sounds like if I expand the retention time a lot, say 30 days the issue > should be less bad? > > > > Thanks > > Shawn > > > > *From: *Mark Payne <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Monday, August 10, 2020 at 12:37 PM > *To: *"[email protected]" <[email protected]> > *Subject: *Re: Data Provenance Stops Working > > > > Shawn / Wyll, > > > > I think you’re probably running into NIFI-7346 [1], which basically says > there’s a case in which NiFi may “age off” old data even when it’s still > the file that’s being actively written to. In Linux/OSX this results in > simply deleting the file, and then anything else written to it disappears > into the ether. Of course, now the file never exceeds the max size, since > it’ll be 0 bytes forever, os it never rolls over again. So when this > happens, no more provenance data gets created until NiFi is restarted. > > > > It’s also possible that you’re hitting NIFI-7375 [2]. This Jira only > affects you if you get to provenance by right-clicking on a Processor and > clicking View Provenance (i.e., not if you go to the Hamburger Menu in the > top-right corner and go to Provenance from there and search that way). If > this is the problem, once you right-click and go to View Provenance, you > can actually click the Search icon (magnifying glass) in that empty > Provenance Results panel and click Search and then it will actually bring > back the results. So that’s obnoxious but it’s a workaround that may help. > > > > The good news is that both of these have been addressed for 1.12.0, which > sounds like it should be coming out very soon! > > > > Thanks > > -Mark > > > > [1] https://issues.apache.org/jira/browse/NIFI-7346 > > [2] https://issues.apache.org/jira/browse/NIFI-7375 > > > > > > On Aug 10, 2020, at 1:26 PM, Joe Witt <[email protected]> wrote: > > > > shawn - i believe it is related to our default settings and have phoned a > friend to jump in here when able. but default retention and default > sharding i *think* can result in this. You can generate a thread dump > before and after the locked state to see what it is stuck/sitting on. That > will help here > > > > Thanks > > > > On Mon, Aug 10, 2020 at 10:24 AM Shawn Weeks <[email protected]> > wrote: > > Out of the box even the initial admin user has to be granted permission I > think, mine worked fine for several months since 1.11.1 was released and > just started having an issues a couple of weeks ago. I’ve increasing the > retention time a bit to see if that improves the situation a bit. > > > > Thanks > > Shawn Weeks > > > > *From: *Wyll Ingersoll <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Monday, August 10, 2020 at 12:22 PM > *To: *"[email protected]" <[email protected]> > *Subject: *Re: Data Provenance Stops Working > > > > I run 1.11.4 in a cluster on AWS also and have a similar issue with the > provenance data, I can't ever view it. It's probably somehow misconfigured > but I haven't been able to figure it out. > ------------------------------ > > *From:* Andy LoPresto <[email protected]> > *Sent:* Monday, August 10, 2020 1:11 PM > *To:* [email protected] <[email protected]> > *Subject:* Re: Data Provenance Stops Working > > > > Shawn, > > > > I don’t know if this is specifically related, but there were a number of > critical issues discovered in the 1.11.x release line that have been fixed > in 1.11.4. I would not recommend running any prior version on that release > line. > > > > 1.12.0 should be coming imminently, so if you are going to upgrade anyway, > you may want to wait a week or so and get the newest bits with hundreds of > new features, but for stability alone, I would strongly recommend 1.11.4. > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes > > > > > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > He/Him > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > On Aug 10, 2020, at 10:00 AM, Shawn Weeks <[email protected]> > wrote: > > > > I’m running a three node NiFi Cluster on AWS EC2s using integrated > Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is > 1.8.0_242. For some reason after a period of time Data Provenance goes > blank, old records are no longer queryable and new data provenance doesn’t > appear to get written. No Lucene or other exceptions are logged and > restarting the node causes data provenance to go back to being written > however old data provenance does not re-appear. No exceptions appear when > querying data provenance. All tests have been run as the initial admin user > and roles and permissions appear to be correct. I’ve also checked selinux > audit logs to see if something is being blocked but nothing appears. > > > > WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 > seconds for rollover, 100mb rollover size, 2 query threads, 2 index > threads, compress on rollover, and don’t sync always. > > > > Thanks > > Shawn Weeks > > > > >
