Re: Nifi provenance indexing throughput if it is being used as an event store

Ali You certainly can and at the rates you mention you should be able to keep it for a good while. Just set the properties you need for your system and measure the rate at which prov storage fills. Thanks On Fri, Feb 15, 2019 at 10:29 PM Ali Nazemian wrote: > I didn't mean to use Nifi

Re: Nifi provenance indexing throughput if it is being used as an event store

I didn't mean to use Nifi provenance search for an external provenance search. I meant to use it for internal search provenance but keep the provenance for a longer time than usual. It means instead of expecting it to keep provenance data for a few days, use it as an event store as it also

Re: Why RedisDistributedMapCacheClientService does not support clustered mode redis?

> There is nothing "distributed" about them. That is not true of at least the HBase distributed map cache client. Never used CouchDB, but I believe that is clusterable too. On Thu, Feb 14, 2019 at 8:15 AM Boris Tyukin wrote: > I am not NiFi dev, but personally, after looking at

Re: Automate NiFi Ranger Policies

Hi Chad, I've never done this, but if I were to go about it I would create a script / cron job to poll the NiFi REST API [1] periodically, and upon detection of a new "Application PG", create the corresponding policies in Ranger via its REST API [2]. You'll have to create service accounts in

Re: Automate NiFi Ranger Policies

Kevin, Thanks for the high level thought process. Seems like a feasible solution. Do you know if I would be able to get the user who created the "Application PG" to add them to the Ranger policy so they don't lose access to their own application? Does NiFi keep that information? Thanks, Chad On

Re: Nifi provenance indexing throughput if it is being used as an event store

NiFi provenance searches are not a good integration pattern for external systems. I.e. using it to periodicaly fetch history burdens the cluster (those searches can be heavy) and disrupt normal processing SLAs. Pushing provenance events out to an external system (pitebtially even filtered down to

Automate NiFi Ranger Policies

We use Ranger with NiFi for security and we are looking to automate the creation of our Ranger policies. The way we organize our flows is like this: NiFi Root Canvas > Ingest Channel PG > Application PG We create 3 Ranger Policies per Application PG: -/process-groups/ -/data/process-groups/

S3-compatible services with NiFi

Anyone tried to connect NiFi to something that is API-compatible with S3 like Minio, SWIFT or Ceph? Thanks, Mike

Re: Apply Zookeeper ACL to Existing NiFi Cluster

Ryan, Sorry for the late reply. Are you still looking for a way to do this? If I understand correctly what you're trying to do, you should be able to use the zk-migrator tool to do this. I haven't done this personally, but here is a rough outline of steps you can follow: - Stop the flow on

Re: running multiple commands from a single ExecuteStreamCommand processor

Thanks Mark, that was very helpful. I have chained the commands now and writing the final output to stdout. Thanks again, Vijay > On Feb 13, 2019, at 2:34 PM, Mark Payne wrote: > > Vijay, > > No worries, this thread is fine. The processor will stream the contents of > the FlowFIle to the

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

Sounds great Koji, thank you for looking into that. I'm trying some tests with changes in GzipHandler included methods, will update if i have any useful info from that. patw On Fri, Feb 15, 2019 at 3:39 AM Koji Kawamura wrote: > Hi Pat, > > Thanks for sharing your insights. > I will try

Re: InvokeScriptedProcessor initialize method

Hi Matt, If we are using Groovy language Scripted processor, is there a work around that we could use to call onStopped or close opened connections? Thank you, Sent from Outlook From: Sri Harsha Chavali Sent: Thursday, February 14,

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

Hi Pat, Thanks for sharing your insights. I will try benchmarking before and after "gzip.setExcludedPath()" that Mark has suggested if it helps improving S2S HTTP throughput. Koji On Fri, Feb 15, 2019 at 9:31 AM Pat White wrote: > > Hi Andy, > > My requirement is to use https with minimum tls