Re: [Bug] Duplicate Flow Import From Registry
Eric, This is simply NiFi on your laptop (single node?) talking to a Registry running on some IaaS infrastructure? There is no load balancer/proxy/etc.. in between ? Thanks On Thu, Dec 10, 2020 at 4:17 PM Eric Secules wrote: > Hello everyone, > > My team is encountering a bug where we import a flow from our registry > residing in the cloud to our laptops. The import takes a long time and we > end up with multiple copies (about 6) of the same flow one on top of each > other on the canvas. The canvas becomes unresponsive and we are unable to > move boxes around. > > I am also experiencing slowness when showing local changes, reverting > local changes and changing the local version. > > Thanks, > Eric >
Re: Need help setting up NiFi PutS3Object processor
Dennis,
You can add user-defined metadata via dynamic properties of PutS3Object
itself (no UpdateAttribute needed). Plus (+) icon at the top-right corner
on the processor properties tab.
The name of the dynamic property will be the key (or name) of the
user-defined metadata.
The value of the dynamic property will be the value of the user-defined
metadata. You can use expression language for the value (eg. referencing FF
attributes or now() for the timestamp).
Hope it helps.
Best,
Peter Turcsanyi
On Thu, Dec 10, 2020 at 11:27 PM Dennis N Brown wrote:
> Running NiFi v1.12.1 , attempting to copy files from existing filesystem
> to S3.
>
> Trying to set user-defined metadata on the file objects, but can't seem to
> figure out the correct format.
>
> All the doc suggests it can be done with dynamic properties, so have an
> UpdateAttribute processor before the PutS3Object processor and have
> attempted everything I can think of to pass the correct format, but no
> metadata gets created.
>
> I have been able to set metadata with python using something like:
>
> *s3.upload_file(Filename, bucketname, key, ExtraArgs={"Metadata":
> {"upload_ts":"2020-12-02 10:57:00.34343","path":"put/some/path/here"}})*
>
> so I know it can be done.
>
>
>
> Can someone provide an example of the correct format to use.
>
>
>
> Thanks!
>
>
>
>
>
> Regards,
>
>
>
> Dennis N Brown
>
>
>
>
>
>
>
>
>
>
>
[Bug] Duplicate Flow Import From Registry
Hello everyone, My team is encountering a bug where we import a flow from our registry residing in the cloud to our laptops. The import takes a long time and we end up with multiple copies (about 6) of the same flow one on top of each other on the canvas. The canvas becomes unresponsive and we are unable to move boxes around. I am also experiencing slowness when showing local changes, reverting local changes and changing the local version. Thanks, Eric
Need help setting up NiFi PutS3Object processor
Running NiFi v1.12.1 , attempting to copy files from existing filesystem to S3.
Trying to set user-defined metadata on the file objects, but can't seem to
figure out the correct format.
All the doc suggests it can be done with dynamic properties, so have an
UpdateAttribute processor before the PutS3Object processor and have attempted
everything I can think of to pass the correct format, but no metadata gets
created.
I have been able to set metadata with python using something like:
s3.upload_file(Filename, bucketname, key, ExtraArgs={"Metadata":
{"upload_ts":"2020-12-02 10:57:00.34343","path":"put/some/path/here"}})
so I know it can be done.
Can someone provide an example of the correct format to use.
Thanks!
Regards,
Dennis N Brown
Re: How to disconnect rabbitmq in PublishAMQP
Hi Franco, If I understand your use case correctly, you do not ingest data continuously and would like PublishAMQP to disconnect after an idle period (when there are no messages to send to RabbitMQ). PublishAMQP is rather optimized for continuous / frequent data load and for this reason it pools the connection(s) and only disconnects when it gets stopped. Configuring idle timeout is not possible currently. The processor holds 1 connection per concurrent task so you can minimize the open connections by setting Concurrent Tasks = 1 (default) on the processor. Best, Peter On Thu, Dec 10, 2020 at 3:48 AM walter franco wrote: > Hello, > I have a question and I need your help: > > I have a project (PoC) using apache nifi with PublishAMQP processor, > writing messages in RabbitMQ, and I see that the connections stay ACTIVE > always, don´t disconnect never (only stop the PublishAMQP processor). The > RabbitMQ has limite of the connections, so I need that diconnect in X > seconds or minutes. > > Is this possible disconnect? or How is it done to free the connections? > > Thanks > > Best regards, > Franco >
ExecuteScript Concurrent Tasks
Hello, Concurrent tasks increased using ExecuteScript or InvokeScriptedProcessor with python/jyphon to update content has no increased throughput. If I copy the processor and run the 2 in parallel the amount of data processed does not increase. Any explanation for this? Is there a system-wide setting for how much cpu is available to the Jython engine? Would refactoring into groovy improve throughput or is it best to create a custom processor? thank you Noe
How to debug 500 error in nifi-api /controller-services method
I set the root logger to debug in logback.xml, and I'm still not seeing any stacktraces in nifi-app.log. Is there something else I need to update? Thanks, Mike
Re: Utilizing NetflowV9 in NiFi v1.12.1
Hi IMHO, since most of the "interesting" fields in NetflowV9 are somehow Vendor-specific, you'll be unlikely to find a generic NiFI-Processor, complete and ready-to-use. We ended up writing a NiFi processor from scratch with different "dialect" settings per Netflow-generating device family, only implementing the decoding needed in our use case. The repo linked in Pierre's answer looks like a very good starting point though. On Tue, Dec 8, 2020 at 8:43 PM Thane Durey wrote: > > Hello all, > > Has anyone found a solution to utilize NetflowV9? I know NiFi v1.12.1 has a > processor to parse Netflow version 5, but I haven't found any information on > Netflow version 9. > > From all the browsing I've done, I've only found a few tickets/issues that > discuss it in the issues.apache.org/jira page. But I don't know if it's > actively being worked on, or when it will be released > > Has anyone implemented anything v9? > > Any help would be appreciated, > > Thanks, > T
GUI functions not available in secure / cluster instance
Hello all. Strange behaviour on the GUI. For background, I have 3 nodes in cluster. Authentication based on LDAP. Groups based also on LDAP. In nifi.properties nifi.security.allow.anonymous.authentication=false And with this, when I want to access the advanced setting of UpdateAttribute : [image: image.png] In web console, I can see permission denied on ressources: [image: image.png] Same when I want to display the flowfile content : [image: image.png] I must put in nifi.properties nifi.security.allow.anonymous.authentication=true To have access to those functions. So in my understanding, when I have secure instance, I must allow anonymous authentication. Etienne
Re: ForkRecord 1.12.0 duplicates flowfiles
No worries, glad you sorted it out and let us know. Pierre Le jeu. 10 déc. 2020 à 11:59, muneti a écrit : > Thanks, Pierre. > > When I have reproduced the behaviour in a simple template, i have realized > the mistake was mine: I had two overlapped success queues at the fork > processor input!! > > Thanks and sorry for the inconvenience. > > Guillermo. > > > > -- > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/ >
Re: ForkRecord 1.12.0 duplicates flowfiles
Thanks, Pierre. When I have reproduced the behaviour in a simple template, i have realized the mistake was mine: I had two overlapped success queues at the fork processor input!! Thanks and sorry for the inconvenience. Guillermo. -- Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: Utilizing NetflowV9 in NiFi v1.12.1
Hi, I'm aware of some people using NiFi for that but I believe they're using NARs available from external Github repositories. We don't have anything packaged OOTB in NiFi itself. For example https://github.com/simonellistonball/nifi-netflow Thanks, Pierre Le mar. 8 déc. 2020 à 20:43, Thane Durey a écrit : > Hello all, > > Has anyone found a solution to utilize NetflowV9? I know NiFi v1.12.1 has > a processor to parse Netflow version 5, but I haven't found any information > on Netflow version 9. > > From all the browsing I've done, I've only found a few tickets/issues that > discuss it in the issues.apache.org/jira page. But I don't know if it's > actively being worked on, or when it will be released > > Has anyone implemented anything v9? > > Any help would be appreciated, > > Thanks, > T >
