Provenance Event Use Cases?

liance requirements? Or data governance issues? Such problems however generally need information at the _flow_ level, not at the per-message level.. Thanks in advance, Simon

Provenance Event performance

, generating 1 million provenance events (or worse) is not helpful.. Thanks in advance, Simon

provenance events and performance

, generating 1 million provenance events (or worse) is not helpful.. Thanks in advance, Simon

Re: Provenance Event Use Cases?

38948494/what-is-the-purpose-of-data-provenance-in-apache-nifi-processors (5) Regarding communication/explanations: yes, more documentation is always needed :-). I do think a section in the NiFi user guide on "why NiFi" would be useful, and would benefit from some information on the use

Re: Provenance Event performance

t tracking each one is simply not practical. In addition, some datasources hold very confidential data; it doesn't seem desirable to record this at all - although AFAIK avoiding retaining this data in the NiFi content repository for unknown periods of time is unavoidable.. Thanks once again

Peers list empty for older Nifi instances

ove make sense? Any advice on a quick fix? Can you specify a peer persistence file like with s2s so it doesn't need to check the older versions instances? Cheers, Simon

Error instantiating template on cluster: The specified observer identifier already exists.

anyone seen this before? Any ideas on why I may be getting this error? Thank you, Simon

Re: How to extract attributes from ListenHTTP flow file?

you down the line, since I’m guessing you’ll want to do things like EvaluateJsonPath on the content of the entry object for different routing for example. A hybrid of b and a is also possible through c. Hope that helps, Simon On 2 Jan 2016, at 11:40, BOUKRAA, Kacem mailto:bk_bouk...@esi.dz

Re: Is it possible to to create a dynamic Processor through API call from PHP

Why not use the SSL layer provided by ListenHTTP? Simon On 2 Jan 2016, at 08:47, BOUKRAA, Kacem mailto:bk_bouk...@esi.dz>> wrote: Hello, I'm using a php script because the source requires an ssl connexion which is not the case with Nifi connexions. I've set up a listene

Re: Deploying flows

You can do that by exporting templates, and then importing those to your prod environment and deploying them on that canvas. Note that this also means sensitive values will need to be reentered, which is usually a good thing when moving to a different environment. Simon On 11 Jan 2016, at 13

Re: PAM Based Authentication & Kerberos

Note that you can certainly write to a kerberised Hadoop cluster at the moment, but you have to provide a principal and key tab file in the PutHDFS config, so you can definitely use nifi with a Kerberos enabled cluster today, you just won't be able to impersonate an end user. Simon On

Re: Thread Control of Processors

event requests queuing behind the InvokeHttp. Feeding failure retries back into the ControlRate, or a funnel before the invoke also gives you a bit more control over groupings of retries for example. Simon Sent from my iPhone - Simon Elliston Ball Solutions Engineer - EMEA +44 7930 424111 Hortonworks

Re: Nifi JSON event storage in HDFS

into hdfs. Avro also makes a lot of sense and can be generated directly out of nifi. Simon - Simon Elliston Ball Product Solutions Architect +44 7930 424111 Hortonworks - Powering the future of data On 2 Mar 2016, at 11:33, Mike Harding mailto:mikeyhard...@gmail.com>> wrote: Hi All,

Re: Aw: Re: central place to collect templates

There is a collection of templates for common patterns at https://github.com/hortonworks-gallery/nifi-templates Feel free to submit any interesting use cases as PRs to that GitHub. Simon On 24 Mar 2016, at 17:50, Uwe Geercken mailto:uwe.geerc...@web.de>> wrote: Joe yes I know this on

Re: Sqoop Support in NIFI

more detail on your use case, and why you needed sqoop on top of NiFi. Simon On 29 Mar 2016, at 09:06, prabhu Mahendran mailto:prabhuu161...@gmail.com>> wrote: Hi, Yes, In my case i have created the Custom processor with Sqoop API which accommodates complete functionality of sqoop. As p

Re: Sqoop Support in NIFI

to ORC format so convert the Avro data to ORC within NiFi. Simon On 30 Mar 2016, at 07:06, prabhu Mahendran mailto:prabhuu161...@gmail.com>> wrote: For Below reasons i have choose Sqoop in NIFI Processor is the best method to move data in Table Structure. If once move the Tabl

Re: Large dataset on hbase

account for java object overhead. The other thing to check is the buffer sizes you are using for your replace text processors. If you’re also using Split processors, you can sometime run up against RAM and open file limits, if this is the case, make sure you increase the ulimit -n settings. Simon On

Re: Apache NiFi/Hive - store merged tweets in HDFS, create table in hive

per line) and then just using binary concatenation in the MergeContent processor Bryne mentioned. Simon On 21 Apr 2016, at 22:38, Bryan Bende mailto:bbe...@gmail.com>> wrote: Also, this blog has a picture of what I described with MergeContent: https://blogs.apache.org/nifi

Re: Apache NiFi - WebHDFS

a datanodes, original request to namenode). This is not strictly speaking correct http behaviour, so might be harder to implement with InvokeHttp. On that basis I would probably vote specific processor. Simon On 22 Apr 2016, at 01:37, Jeremy Dyer mailto:jdy...@gmail.com>> wrote: Yep

Re: GetMongo question

to maintain latest timestamps. Given the way Mongo ObjectIDs embed a timestamp, this could be done quite neatly. Would you be interested on working on contributing something like that? The first step would be to put in an enhancement ticket at https://issues.apache.org/jira/browse/NIFI Simon

Re: Question on setting up nifi flow

store them in NiFi’s cache. The main ingest flow would then use UpdateAttributes to create a hash.value that matched the values inserted to the cache -> DetectDuplicates -> flow to PutMongo (use the upset property) -success-> PutSolrContentStream Simon On Apr 28, 2016, at 5:19 PM, Pierr

Re: Sensitive values

in the template it would not be portable. A possible solution would be to use the template to deploy, and then an API call to update the sensitive values. Simon > On May 4, 2016, at 8:37 AM, Ravisankar Mani wrote: > > Hi All, > > I have created a specific template. In this te

Re: NiFi app log

The log rotation is controlled by the logback.xml in conf/ See http://logback.qos.ch/manual/configuration.html for instructions on how to control rotation. Simon On 6 May 2016, at 17:40, Igor Kravzov mailto:igork.ine...@gmail.com>> wrote: Is there a configuration option to produce

flow.xm.gz not being archived to ./conf/archive in NiFi 0.6.1

solution by searching the archives. My nifi.properties is below. I am running on CentOs 7. Am I missing something obvious? Shouldn't my flow.xml.gz file be archived into ./conf/archive whenever a new flow.xml.gz file is generated? Thanks, Simon # Core Properties # nifi.version=

Re: flow.xm.gz not being archived to ./conf/archive in NiFi 0.6.1

Mark, That explains it. Thank you. Simon -Mark Payne wrote: - To: users@nifi.apache.org From: Mark Payne Date: 10/03/2016 11:26AM Subject: Re: flow.xm.gz not being archived to ./conf/archive in NiFi 0.6.1 Simon, The ability to have NiFi automatically back up the flow was added

Use AWS:KMS SSE with PutS3Object?

, i.e.: aws s3 cp --sse aws:kms --sse-kms-key-id Thanks, Simon

RE: Re: Use AWS:KMS SSE with PutS3Object?

James, OK, thank you very much for the reply and the Jira link. Simon From: James Wing [mailto:jvw...@gmail.com] Sent: Wednesday, April 18, 2018 2:30 PM To: users@nifi.apache.org Subject: [External] Re: Use AWS:KMS SSE with PutS3Object? Simon, I'm afraid NiFi does not yet support server

Australian Apache NiFi Experts

Looking for an Australian based Apache NiFi resource for a small project to POC some aspects of the product . Posted on the Slack community and was advised to post here . -- Simon Sigré simon.si...@gmail.com https://simonsigre.com | @simonsigre

Re: Split JSON using an expression to define the PATH

Hi, Without knowing the actual use case: what if you would run the JSON through JoltTransformJSON and convert it into a format would work well with the SplitJson? It might help. (On the side note: as you mention “huge” JSON, it might be a resource consuming operation) Regards, Bence > On 2020

Re: Round robin load balancing eventually stops using all nodes

Hi Mike, I did a quick check on the round robin balancing and based on what I found the reason for the issue must lie somewhere else, not directly within it. The one thing I can think of is the scenario where one (or more) nodes are significantly slower than the other ones. In these cases it mi

Re: NiFi component status history not showing more than a days worth of data

Hi Michael, I did not have time to take a deeper look on this, but I had a fix for cleanup related issues lately: https://github.com/apache/nifi/pull/6374 As for the size of the stored data, it might be perfectly normal. The amount of data primarily d

Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows

Hi Josef, Thank you for raising the attention to this. The storageLocation is a new concept to generalise the information used to locate nested flows in a versioned flow. In general it can be there and does not cause issues, as of now NiFi falls back the previous way to locate nested flows. As

Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows

://issues.apache.org/jira/browse/NIFI-10874 <https://issues.apache.org/jira/browse/NIFI-10874> which is not part of NiFi 1.19.0. The next release should contain the fix that addresses your issue. Regards, Bence > On 2022. Nov 28., at 17:48, Simon Bence wrote: > > Hi Josef, >

Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows

s and suddenly one of the other nested > flows have been losing it’s storageLocation in the parent flow and from this > point the whole flow was broken. Now we are scared to change anything as it > could brake anytime again. > > Cheers Josef > > > > Fr

Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows

tps://github.com/apache/nifi/pull/6741> I would not recommend to go forward without this additional change! Regards, Bence > On 2022. Nov 29., at 16:21, Simon Bence wrote: > > Hi Josef, > > Your welcome! In your case it looks like the information is lost during > transfo

Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue

Hi Josef, Thanks for the quick update! Meanwhile I tried to reproduce this situation but based on the known facts I was not able (Note: without any version change). I think it is connected to the version change indeed, but if you face with this ever again, please let me know! Regards, Bence >

Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue

jira/browse/NIFI-10973> with more details. It’s as > well related to NiFi Registry nested flows… > > Thanks for taking care of it :-). > > Cheers Josef > > > From: Simon Bence mailto:simonbence@gmail.com>> > Reply to: "users@nifi.apache.org <ma

Re: Embedded Hazelcast Cachemanager

about the error and possibly the intended way of usage, it could help to find a more specific answer. Best regards, Bence Simon > On 2023. Feb 20., at 15:19, Isha Lamboo > wrote: > > Hi all, > > This morning I had to fix up a cluster of NiFi 1.18.0 servers where

Re: Embedded Hazelcast Cachemanager

sible optimization opportunity to distribute the batch in time resulting smaller peaks. Best regards, Bence > On 2023. Feb 21., at 21:45, Isha Lamboo > wrote: > > Hi Simon, > > The Hazelcast cache is being used by a DetectDuplicate processor to cache and > eliminate message ids. T

Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue

pened a Jira Bugticket > https://issues.apache.org/jira/browse/NIFI-10973 with more details. It’s as > well related to NiFi Registry nested flows… > > Thanks for taking care of it :-). > > Cheers Josef > > > From: Simon Bence > Reply to: "users@nifi.ap

Re: NiFi 1.19.1 Registry Nested Process Groups - Show Local Changes issue

Hi Josef, There is a PR made by Mark takes care of this behaviour: https://github.com/apache/nifi/pull/7095 I hope this will fit to your needs! If there is something is going unexpected, please reply! Regards, Bence > On 2023. Mar 28., at 9:14, Simon Bence wrote: > > Hi Josef, &g

Re: HMAC key storage?

src/main/java/org/apache/nifi/processors/standard/GetHTTP.java#L172> for an example of how this works. Simon > On 18 Dec 2016, at 09:32, Jonathan Telfer wrote: > > Hi all > > I'm currently writing a custom processor to access a non standard messaging > protocol. Th

Re: Netflow parser

awesome in NiFi. Simon > On 17 Apr 2017, at 08:34, Joe Witt wrote: > > I've seen several over the past couple years used in NiFi. I'd bet > with a little bit of work and collab with Apache Metron we could have > one quickly should someone be in a position to contribute. >

Re: Use Case...Please help

that drive with its windows credentials. There isn’t currently a means to provide authentication per share in the processor, but nifi should inherit the credential context of whichever user is running the nifi process. Hope that helps, Simon > On 22 May 2016, at 18:40, Tripathi, Shiv Dee

Re: GetHTTP->ExtractText (Regex/User problem?)

ng, but will certainly save you a lot of problems with parsing. This lets you use css selectors against html, which is more intuitive and robust to parse HTML. Simon > On 20 Jun 2016, at 18:43, Sven Davison wrote: > > I had tried that but got a NULL value result. Is there a setting w/in th

Re: GetHTTP->ExtractText (Regex/User problem?)

If you use GetHTMLElement, and set the destination property to flow file-attribute, that will store the content as an attribute flow file you can use later in expression language for example. Is that what you're after? Simon > On 20 Jun 2016, at 19:50, Sven Davison wrote: > >

Re: QRadar

Yes, the best way I’ve seen is via a syslog pipe out of Qradar into Nifi (which then went on to use Apache Metron and parsed using their LEEF parser). It is also possible to hook qradar up direct to Kafka, but that I believe (don’t quote me on this) is an optional extra. Simon On Wed, 4 Sep 2019