Re: Atlas and NiFi integration help

Bryan, That did it. It might not be able to answer at the granularity of how many times a reindex was done, but with the right mix of updateattributes and that task, I was able to build an ElasticSearch index that can at least show a date histogram aggregation of when reindex operations happened

Re: Atlas and NiFi integration help

Mike, That is basically the point of SiteToSiteProvenanceReportingTask... you send the provenance events from reporting task back to the same cluster, and then leverage existing processors like the ElasticSearch processors. Otherwise we'd get into building 100 reporting tasks for all the various

Re: Atlas and NiFi integration help

Bryan, I have a feeling you're right. This might call for a reporting task that exports to ElasticSearch so that Kibana dashboards can be used to answer these questions. Thanks, Mike On Thu, Mar 1, 2018 at 10:20 AM, Bryan Bende wrote: > Mike, > > As far as I know, Atlas is

Re: Atlas and NiFi integration help

Mike, As far as I know, Atlas is not really about "event level" lineage, it is more about "system level" or "data set' level. So I believe the goal of Atlas is to show how the systems are connected and how a particular data set flows through the system. So an example might be... NiFi pulls from

Re: Atlas and NiFi integration help

Matt, Yeah, I saw that pretty early on. Admittedly my question may be a bit nebulous. What I'm trying to figure out is what I should be seeing in Atlas if NiFi is sending it events properly. Since the integration and knowledge around it is probably clustered here, I'm not sure I can go to the

Re: Atlas and NiFi integration help

Mike, There is a nifi-atlas-bundle in NiFi with a NAR that includes the ReportLineageToAtlas reporting task, but IIRC it is so large that it is not included in the default assembly. Instead there is a "include-atlas" profile that can be activated when building the assembly, and that should

Atlas and NiFi integration help

I have Atlas 0.8.2 (BerkeleyDB and Embedded ES) and NiFi 1.6.0 nightly both up and claiming that they can talk to one another. What should I be seeing if they are? My test configuration consists of a simple process group that has GetMongo, UpdateAttributes and PutElasticSearchHttpRecord. I'm not