Re: NIFI HandleHttpRequest API - Health Check when API or Node Down

I do not know everything, but if I well understood NiFi is based on REST API. For example, all you do on the GUI is done throw REST call. So I guess you can request if the NiFi instance is up on each node. But this will not give you the status of your custom HttpHandle. NiFi instance can be up,

Re: NIFI HandleHttpRequest API - Health Check when API or Node Down

You can always hit NiFi API status rest endpoint. It won't give you any idea about that specific http endpoint you exposed, though, as it is a general nifi rest api. Your LB would need to understand how to hit this URL too, especially if it's secured. Coming back to the easiest path, you'd rather

Re: NIFI HandleHttpRequest API - Health Check when API or Node Down

It seems a bit like a chicken and egg thing. Using ‘anything’ configured on the disconnected node as a health check, is not unlike trying to get to the API (listening port) itself? Kinda. Anyway I was hoping that the NIFI infrastructure had a generalized, centralized (REST API? or other)

Re: NIFI HandleHttpRequest API - Health Check when API or Node Down

Because you implemented a HandleHttpRequest listing, why don't you configure an handle on something like http(s)://server/ping And the response is just pong Le ven. 4 sept. 2020 à 18:02, jgunvaldson a écrit : > Hi, > > Our network administrators are unable to wire up advanced Load Balancer >

Re: Nifi security of local filesystem and hdfs in multitenant hdfs use cases

We have developed integrations with Hashicorp Consul and Vault, partly to deal with this same use-case. I'm happy to open source these if there is an interest. On Fri, 28 Aug 2020, at 17:25, oliver twix wrote: > I would be great I think. > > Le ven. 28 août 2020 à 17:20, Bryan Bende a écrit

Re: Nifi security of local filesystem and hdfs in multitenant hdfs use cases

I would be great I think. Le ven. 28 août 2020 à 17:20, Bryan Bende a écrit : > The reason for requiring the FS permissions on the HDFS processors is > because you can provide a core-site.xml with file:/// as the default FS and > then use ListHDFS/FetchHDFS/PutHDFS to essentially do the same

Re: Nifi security of local filesystem and hdfs in multitenant hdfs use cases

The reason for requiring the FS permissions on the HDFS processors is because you can provide a core-site.xml with file:/// as the default FS and then use ListHDFS/FetchHDFS/PutHDFS to essentially do the same thing as ListFile/FetchFile/PutFile. A possible consideration would be to remove the

Re: Nifi security of local filesystem and hdfs in multitenant hdfs use cases

Hello, thank you for your quick answers and sorry for my late reply. In the context of preparing platform security audit, I am trying to identify possible threats over our NIFI clusters. @Mark, I didn't know about NIFI_ALLOW_EXPLICIT_KEYTAB parameter. Good to know; it will limit a semi-friendly

Re: NIFI - Load Balanced APIs

John, If you’re running a proxy in front of NiFi, you will definitely need to configure some sort of session stickiness. Otherwise you can run into other issues as well. For instance, some "asynchronous requests” like updating variables, etc. may intermittently fail because the request creates

Re: Nifi takes too long to start(~ 30 minutes)

Hi Andy, Apologies for the late reply. Yes there are around 25000 processors (1000 flows with around 25 processors each, I mistyped.) I started the nifi in the debug mode to get something but nothing concrete. Following logs gets printed for all the processors: 2020-08-24 22:56:04,416 INFO

Re: NiFi 1.12.0 - KeyStores with multiple certificates are not supported

2020 at 01:06 To: "users@nifi.apache.org" Subject: Re: NiFi 1.12.0 - KeyStores with multiple certificates are not supported Hi Josef and Kotaro, Thanks for identifying this scenario. I am away from the office for a bit but will try to review Kotaro’s changes in the linked PR. The regression

Re: NiFi 1.12.0 - KeyStores with multiple certificates are not supported

Hi Josef and Kotaro, Thanks for identifying this scenario. I am away from the office for a bit but will try to review Kotaro’s changes in the linked PR. The regression is within Jetty’s code, and requires a new API to be invoked. NiFi does not have an existing method to configure a specific

Re: NiFi equivalent to Java's finally()?

Off the top of my head, based on what you are describing, I would consider just using a funnel. Your flow would start somehow, then maybe slit based on some logic, and if you wanted everything to come back together, just create a funnel that leads back to your final processor / set of processors.

Re: Nifi takes too long to start(~ 30 minutes)

Mohit, I’m a little confused by some of the details you report. Initially you said there were ~100 flows with 25 processors each (2500); now there are 25,000. If you examine the logs, you note that “reaching the election process” takes the majority of the time — what messages are being printed

Re: Nifi takes too long to start(~ 30 minutes)

Hi Pierre, No, the election process takes only a minute or two max, reaching the election process takes a lot of time. There are around 25k stopped/invalid components in the UI, I marked all of them as disabled hoping it would resolve the issue but no improvement in the startup time. On a

Re: Nifi takes too long to start(~ 30 minutes)

> > Brandon > > -- > *From:* Pierre Villard > *Sent:* Thursday, August 13, 2020 12:10:51 PM > *To:* users@nifi.apache.org > *Subject:* Re: Nifi takes too long to start(~ 30 minutes) > > Hi, > > I'm surprised this is something you observe in the ele

Re: Nifi 1.11.4 three node cluster is taking longer time to up after migration from 1.8

Thanks Mark for the update, I will use Dynatrace to analyse the heap memory and also look for the flows as you suggested. On Fri, Aug 14, 2020 at 7:40 PM Mark Payne wrote: > Sanjeet, > > It’s hard to say what is triggering the OutOfMemoryError. I was able to > create a similarly sized flow and

Re: Nifi 1.11.4 three node cluster is taking longer time to up after migration from 1.8

Sanjeet, It’s hard to say what is triggering the OutOfMemoryError. I was able to create a similarly sized flow and startup just fine using a 2 GB heap. Since you’ve got 8 GB of heap, it’s unlikely related to the size of just the flow itself. The only way to really definitively diagnose an

Re: Nifi 1.11.4 three node cluster is taking longer time to up after migration from 1.8

Hi Mark/Team Any thoughts on this? Where should i analyse further. Regards, Sanjeet On Thu, 13 Aug 2020, 10:32 pm sanjeet rath, wrote: > Hi Mark, > Thanks for the response. > > My flow.xml.gz file size is 14 Mb. > > Regards, > Sanjeet > > On Thu, 13 Aug 2020, 10:27 pm Mark Payne, wrote: >

Re: Nifi 1.11.4 three node cluster is taking longer time to up after migration from 1.8

Hi Mark, Thanks for the response. My flow.xml.gz file size is 14 Mb. Regards, Sanjeet On Thu, 13 Aug 2020, 10:27 pm Mark Payne, wrote: > Actually, I take back what I said. I was a little too quick to jump to > conclusions about what the issue was. There was an issue addressed that > should

Re: Nifi 1.11.4 three node cluster is taking longer time to up after migration from 1.8

Actually, I take back what I said. I was a little too quick to jump to conclusions about what the issue was. There was an issue addressed that should improve startup time. But what you’re seeing here is unrelated, as you’re enchanting OutOfMemoryError. The long time is likely related to garbage

Re: Nifi 1.11.4 three node cluster is taking longer time to up after migration from 1.8

Sanjeet, I believe this should be addressed in 1.12.0, which should be released very soon. Thanks -Mark On Aug 13, 2020, at 4:13 AM, sanjeet rath mailto:rath.sanj...@gmail.com>> wrote: Hi Team, I have migrated my flows.xml.gz, users.xml, authorization.xml from 1.8 env to 1.11.4 environment

Re: Nifi takes too long to start(~ 30 minutes)

> > Brandon > > -- > *From:* Pierre Villard > *Sent:* Thursday, August 13, 2020 12:10:51 PM > *To:* users@nifi.apache.org > *Subject:* Re: Nifi takes too long to start(~ 30 minutes) > > Hi, > > I'm surprised this is something you observe

Re: Nifi takes too long to start(~ 30 minutes)

Mohit, How many flowfiles are currently on the instance? Sometimes a very large number of flowfiles can result in slower start times. Brandon From: Pierre Villard Sent: Thursday, August 13, 2020 12:10:51 PM To: users@nifi.apache.org Subject: Re: Nifi takes

Re: Nifi takes too long to start(~ 30 minutes)

Hi, I'm surprised this is something you observe in the election process part. I've constantly seen quick startup times even with thousands of components in the flow. I'd look into the logs and maybe turn on some debug logs to find out what's going on. Pierre Le jeu. 13 août 2020 à 16:33, Joe

Re: Nifi takes too long to start(~ 30 minutes)

Mohit, You almost certainly want to take that same flow and setup a cluster on a more recent version of NiFi to compare startup times. For flows with thousands of components there are important improvements which have occurred in the past year and a half. Startup time, user perceived behavior

Re: NIFI /Kafka - Data loss possibility with node failures

Thanks Joe. This is really helpful. On Tue, Aug 11, 2020 at 9:33 AM Joe Witt wrote: > Asmath > > In a traditional installation, regardless of how a NiFi cluster obtains > data (kafka, ftp, HTTP calls, TCP listening, etc, ) once it is > responsible for the data it has ack'd its receipt to

Re: NIFI /Kafka - Data loss possibility with node failures

Asmath In a traditional installation, regardless of how a NiFi cluster obtains data (kafka, ftp, HTTP calls, TCP listening, etc, ) once it is responsible for the data it has ack'd its receipt to the source(s). If that NiFi node were to become offline the data it owns is delayed. If that node

Re: NiFi Merge Content Processor Use Case

I have created a JIRA issue for this: https://issues.apache.org/jira/browse/NIFI-7699 On Fri, Jul 31, 2020 at 10:20 AM Eric Secules wrote: > Hi Joe, > > Thanks for your suggestions and analysis! > > I was also thinking that the MergeContent processor could also have a wait > relationship so

Re: NiFi Merge Content Processor Use Case

Hi Joe, Thanks for your suggestions and analysis! I was also thinking that the MergeContent processor could also have a wait relationship so that way we can distinguish between new flowfiles and flowfiles that didn't fit into a bucket before. Does NiFi have the ability to distinguish between

Re: NiFi Merge Content Processor Use Case

Eric I believe for now the most appropriate option available is to figure out what the maximum set of outstanding in process buckets could be and set that. Hopefully there is enough memory in the system to handle whatever that worse case could be in terms of flowfiles attributes being held in

Re: Nifi 1.8 to 1.11.4 migration issue

Thanks Mark for the update. Could you please the jira request number on which this issue is fixed. Regards, Sanjeet On Fri, 31 Jul 2020, 12:51 am Mark Payne, wrote: > OK, I suspect you’re probably hitting an issue that was introduced in > 1.11.x related to fingerprint handling. A change was

Re: Nifi 1.8 to 1.11.4 migration issue

OK, I suspect you’re probably hitting an issue that was introduced in 1.11.x related to fingerprint handling. A change was made to improve how some things were handled in the flow fingerprinting logic but resulted in very poor performance in some particular cases. I.e., some flows will load

Re: Nifi 1.8 to 1.11.4 migration issue

Hi Mark, Thanks for responding. Sorry, i can't share the log, Due to our company policy. its keeping printing this 2 warning in nifi-app.log once the server is up this warning is gone. 1-> O.apache.nifi.controler.flocontroler failed to send heart beat due to ;

Re: Nifi 1.8 to 1.11.4 migration issue

Sanjeet, I’d recommend grabbing a thread dump while NiFi is starting up (after it’s been going for 3 minutes or so) and providing that to understand why it’s taking 30 minutes to startup. Specifically, the “main” thread will be of interest. Thanks -Mark On Jul 30, 2020, at 2:04 PM, sanjeet

Re: Nifi 1.8 to 1.11.4 migration issue

Sanjeet, just a heads up as we were in the same boat. We decided to hold off the upgrade because of a critical issue in 1.11.4. not sure if it applies to your environment but we decided to wait for 1.12 more details here https://issues.apache.org/jira/browse/NIFI-7460 On Thu, Jul 30, 2020 at

Re: Nifi 1.8 to 1.11.4 migration issue

Hi , yes i am using external zookeeper 3.5 version.I have already followed all the document you have share . Now i am seeing after waiting around 30 mins the node gets connected after that there is no node disconnection issue. when i restarted , again it is taking 30 min for node connection to

Re: Nifi security of local filesystem and hdfs in multitenant hdfs use cases

in short this case has been really deeply considered and it is a very common usage pattern. We offer a set of policies/controls that let it be well restricted and locked down. But if a user is given too many accesses then yes they can be malicious. Thanks On Thu, Jul 30, 2020 at 9:13 AM

Re: Nifi security of local filesystem and hdfs in multitenant hdfs use cases

Olivier, As Joe mentioned, it may help to further explain the exact scenario that you are concerned about. But what I *think* you are concerned about is the following scenario: - You have several different users developing flows in NiFi. - You want the ability to give User A (and only User A)

Re: Nifi 1.8 to 1.11.4 migration issue

Hi Sanjeet, One thing to do is make sure you're using at least Zookeeper 3.5.5: Migrating from 1.x.x to 1.10.0 > >- The RPM creation mechanism appears broken for both Java 8 and Java >11 binaries if you wanted to build those yourself. Should be resolved in a >later release. >-

Re: Nifi 1.8 to 1.11.4 migration issue

Hi, The error is, O.apache.nifi.controler.flocontroler failed to send heart beat due to ; org.apache.nifi.cluster.protocol.protocalException: Failed marshalling 'HEARTBEAT" protocol message due to javax.net.sslException:Read timed out. Before migration there was no issue in cluster and ckuster

Re: Nifi 1.8 to 1.11.4 migration issue

nifi-app.log file , i am getting constant error. Apache.nifi.controler.FlowControler Failed to send heart beat due to org.apache.nifi.protocal.protocolException On Thu, 30 Jul 2020, 6:54 pm Mark Payne, wrote: > Sanjeet, > > What error are you receiving? > > Sent from my iPhone > > On Jul 30,

Re: Nifi 1.8 to 1.11.4 migration issue

Sanjeet, What error are you receiving? Sent from my iPhone On Jul 30, 2020, at 9:21 AM, sanjeet rath wrote:  Hi Team, Any help on my trailed mail query will be highly appreciated.still i am strgling yo fix yhe issue. In nifi-app.log file , i am getting constant error.

Re: Nifi 1.8 to 1.11.4 migration issue

Hi Team, Any help on my trailed mail query will be highly appreciated.still i am strgling yo fix yhe issue. In nifi-app.log file , i am getting constant error. Apache.nifi.controler.FlowControler Failed to send heart beat due to org.apache.nifi.protocal.protocolEcslception. Thanks & Regards,

Re: NiFi Registry Bundles - Integration into NiFi

Thank you Bryan and Juan Pablo for the quick and very helpful answers! I'm looking forward to the future changes regarding the seamless integration of NAR-files in the NiFi registry with NiFi. Until then I'll stick to my current artifact repository and deployment pipeline. Thanks once more!

Re: NiFi Registry Bundles - Integration into NiFi

Correct, there is currently not auto-unloading. On Mon, Jul 20, 2020 at 3:18 PM Juan Pablo Gardella < gardellajuanpa...@gmail.com> wrote: > Be aware that using auto-load works for adding only. If you remove it from > that directory, it does not remove or update AFAIK, at least, on nifi 1.5.0 > >

Re: NiFi Registry Bundles - Integration into NiFi

Be aware that using auto-load works for adding only. If you remove it from that directory, it does not remove or update AFAIK, at least, on nifi 1.5.0 On Mon, 20 Jul 2020 at 16:10, Bryan Bende wrote: > Hello, > > The plan is to eventually build a seamless experience on the NiFi side > where it

Re: NiFi Registry Bundles - Integration into NiFi

Hello, The plan is to eventually build a seamless experience on the NiFi side where it would obtain the required NARs for a versioned flow, but that work has taken place yet. The ability to store the bundles in registry with an API for retrieving them is the starting point for that. Currently

Re: NiFi 1.11.4 Upgrade

ps and fixing them? We would like to > see if they can be fixed on our side and still go with 1.11.4 deployment. > > Thank you, > Harsha > > Sent from Outlook <http://aka.ms/weboutlook> > -- > *From:* Bryan Bende > *Sent:* Wednesday, Ju

Re: NiFi 1.11.4 Upgrade

go with 1.11.4 deployment. Thank you, Harsha Sent from Outlook<http://aka.ms/weboutlook> From: Bryan Bende Sent: Wednesday, July 15, 2020 9:06 AM To: users@nifi.apache.org Subject: Re: NiFi 1.11.4 Upgrade Hello, Unfortunately this was a bug found after

Re: NiFi 1.11.4 Upgrade

Hello, Unfortunately this was a bug found after 1.11.4 and has been fixed in master, but is not in a release yet. https://issues.apache.org/jira/browse/NIFI-7460 The community is hoping to release 1.12.0 soon though. Thanks, Bryan On Tue, Jul 14, 2020 at 10:15 PM Sri Harsha Chavali <

RE: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

ok, but the old db shows as 2.4M, the new is merely 152K. From: Bryan Bende Sent: Friday, July 10, 2020 2:35 PM To: users@nifi.apache.org Subject: Re: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0 Do you have all the same buckets and flows in the registry UI? If so then I would th

Re: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

something in the steps and if you can point it out, > that’d be great. > > Thank you. > > Roland > > *From:* Bryan Bende > *Sent:* Wednesday, July 8, 2020 4:04 PM > *To:* users@nifi.apache.org > *Subject:* Re: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

RE: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

appears to work fine after I did that. DB size still has me concerned though. I may have fumbled something in the steps and if you can point it out, that’d be great. Thank you. Roland From: Bryan Bende Sent: Wednesday, July 8, 2020 4:04 PM To: users@nifi.apache.org Subject: Re: [EXTERNAL] Re: NiFi

Re: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

, July 8, 2020 3:10 PM > *To:* users@nifi.apache.org > *Subject:* [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0 > > > > Hello, > > > > Older releases can always be found in the apache archive: > > > > https://archive.apache.org/dist/nifi/nif

RE: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

Thank you Bryan. Missed this link. Would you have a recommended path to upgrade from 0.1.0 keeping H2? Thank you Roland From: Bryan Bende Sent: Wednesday, July 8, 2020 3:10 PM To: users@nifi.apache.org Subject: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0 Hello, Older releases can

Re: NiFi registry safe upgrade path from 0.1.0

Hello, Older releases can always be found in the apache archive: https://archive.apache.org/dist/nifi/nifi-registry/ Thanks, Bryan On Wed, Jul 8, 2020 at 2:14 PM Rosso, Roland wrote: > Hi all, > > We are upgrading NiFi from 1.9.2 to 1.11.4. Our 1.9.2 uses NiFi Registry > 0.1.0 still and we

Re: NiFi 1.11.4 -- "Unable to access lib/bootstrap to create bootstrap classloader"

Hi Joseph, It looks like there is an issue with the location of the directories/files. What did you change? Did you solve your issue? Thanks, Pierre Le mar. 30 juin 2020 à 16:44, Joseph Wheeler a écrit : > Hello! > > > > I’m trying to deploy NiFi 1.11.4 to a new environment. After configuring

Re: NiFi-light for analysts

I actually do it manually in docker file: RUN mv /opt/nifi/nifi-current/lib/*.nar /opt/nifi/nifi-current/lib.original/ RUN cp /opt/nifi/nifi-current/lib.original/nifi-avro-nar-$NIFI_VER.nar /opt/nifi/nifi-current/lib RUN cp

Re: NiFi-light for analysts

That would be a fine option for those users who are capable to run maven builds. I think evolving the nifi registry and nifi integration to source all nars as needed at runtime from the registry would be the best user experience and deployment answer over time. Thanks On Mon, Jun 29, 2020 at

Re: NiFi-light for analysts

As far as I can tell, Kylo is dead based on their public github activity. Mark, Would it make sense for us to start modularizing nifi-assembly with more profiles? That way people like Boris could run something like this: mvn install -Pinclude-grpc,include-graph,!include-kafka,!include-mongodb

Re: NiFi-light for analysts

Hi Mark, thanks for the great comments and for working on these improvements. these are great enhancements that we can certainly benefit from - I am thinking of two projects at least we support today. As far as making it more user-friendly, at some point I looked at Kylo.io and it was quite an

Re: NiFi-light for analysts

Hey Boris, There’s a good bit to unpack here but I’ll try to answer each question. 1) I would say that the target audience for NiFi really is a person with a pretty technical role. Not developers, necessarily, though. We do see a lot of developers using it, as well as data scientists, data

Re: NiFi 1.11 - non-heap size

I see. Thanks again! Valentina From: Joe Witt Sent: Thursday, 25 June 2020 17:54 To: users@nifi.apache.org Subject: Re: NiFi 1.11 - non-heap size Since the max is undefined if more memory is needed the size will grow and the current usage may ebb and flow thus

Re: NiFi 1.11 - non-heap size

rsday, 25 June 2020 17:35 > *To:* users@nifi.apache.org > *Subject:* Re: NiFi 1.11 - non-heap size > > -1 in that case means it is on a system or configuration for which the JVM > cannot get that answer. > > Nothing to worry about. > > On Thu, Jun 25, 2020 at 8:33 AM Va

Re: NiFi 1.11 - non-heap size

Hi Joe, Thanks for the quick reply! What about the used and free non-heap - free non-heap is 14 MB only while used is 224MB? Thanks Valentina From: Joe Witt Sent: Thursday, 25 June 2020 17:35 To: users@nifi.apache.org Subject: Re: NiFi 1.11 - non-heap size

Re: NiFi 1.11 - non-heap size

-1 in that case means it is on a system or configuration for which the JVM cannot get that answer. Nothing to worry about. On Thu, Jun 25, 2020 at 8:33 AM Valentina Ivanova wrote: > Hello again! > > I have been looking at the system diagnostics and noticed the values for > non-heap size

Re: NiFi 1.11 - High repository storage usage

Hi Wesley & Harald, Thanks for the quick replies! @Wesley C. Dias de Oliveira I have recently increased these settings due to other reasons, so that might be the primary cause. @Harald You are right, they are on the same drive. I assumed that usage is relative

Re: NiFi 1.11 - High repository storage usage

Hi, Valentina. I've experienced the same issue on an old installation. In my case, it was related to memory usage. The system begins to dump things to the disk when there's no available memory. Have you checked the memory( params? # JVM memory settings java.arg.2=-Xms512m java.arg.3=-Xmx512m

RE: NIFI JMS consuler : AMQP 1.0 trouble

Hi Peter Thanks very much for your help !!! It perfectly works. Have a nice day Olivier De : Peter Turcsanyi Envoyé : lundi 22 juin 2020 23:28 À : users@nifi.apache.org Objet : Re: NIFI JMS consuler : AMQP 1.0 trouble Olivier, The problematic property is the JNDI Provider URL. It seems

Re: NIFI JMS consuler : AMQP 1.0 trouble

iel pendant vos temps non travaillés, il ne > requiert > ni prise de connaissance ni réponse immédiate de votre part. > > *De :* Peter Turcsanyi > *Envoyé :* vendredi 19 juin 2020 14:59 > *À :* users@nifi.apache.org > *Objet :* Re: NIFI JMS consuler : AMQP 1.0 trouble > &g

RE: NIFI JMS consuler : AMQP 1.0 trouble

temps non travaillés, il ne requiert ni prise de connaissance ni réponse immédiate de votre part. De : Peter Turcsanyi Envoyé : vendredi 19 juin 2020 14:59 À : users@nifi.apache.org Objet : Re: NIFI JMS consuler : AMQP 1.0 trouble Hi Olivier, You can use Apache Qpid Proton AMQP client library [1

Re: NIFI JMS consuler : AMQP 1.0 trouble

aillés, il ne requiert ni prise de connaissance ni réponse immédiate de votre part. De : Peter Turcsanyi mailto:turcsa...@apache.org>> Envoyé : vendredi 19 juin 2020 14:59 À : users@nifi.apache.org<mailto:users@nifi.apache.org> Objet : Re: NIFI JMS consuler : AMQP 1.0 trouble Hi Oli

RE: NIFI JMS consuler : AMQP 1.0 trouble

pendant vos temps non travaillés, il ne requiert ni prise de connaissance ni réponse immédiate de votre part. De : Peter Turcsanyi Envoyé : vendredi 19 juin 2020 14:59 À : users@nifi.apache.org Objet : Re: NIFI JMS consuler : AMQP 1.0 trouble Hi Olivier, You can use Apache Qpid Proton AMQP client

Re: NIFI JMS consuler : AMQP 1.0 trouble

Hi Olivier, You can use Apache Qpid Proton AMQP client library [1] as recommended in the ActiveMQ AMQP configuration docs [2]. You'll want to download Qpid JMS (AMQP 1.0) and use the jars from the lib folder of the tarball. The JNDI Initial Context Factory Class needs to be configured:

Re: NiFi Expression Language in UpdateAttribute

Russell, I think it would be fine to include an example like this in the Expression Language Guide. You can submit a PR to add that if you like. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun

Re: NiFi Expression Language in UpdateAttribute

That solves confusion I had after and despite reading the NiFi Expression Language documentation. Having written documentation before, I get why it's never ending and danger-fraught to write "bigger" examples in such documentation, and we resist doing it, but that's what I needed to clear up

Re: NiFi Expression Language in UpdateAttribute

Hello, The EL docs have some good examples: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#ifelse -Bryan On Thu, Jun 11, 2020 at 10:14 AM Russell Bateman wrote: > I wnat to create an attribute, format-type, which I hoped would be set > based on the in-coming

Re: NiFi Expression Language in UpdateAttribute

Your syntax is incorrect. Try this instead. ${filename:endsWith('xml'):ifElse('XML','JSON')} From: Russell Bateman Reply-To: "users@nifi.apache.org" Date: Thursday, June 11, 2020 at 9:15 AM To: NiFi Users Subject: NiFi Expression Language in UpdateAttribute I wnat to create an attribute,

Re: NiFi NARs Included in Docker

On purpose/conscious decision as part of 1.10.0, see the release notes and migration guide [1] 1. https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.10.0 Cheers, Chris Sampson On Mon, 1 Jun 2020, 18:42 Eric Secules, wrote: > Hello everyone, > > I noticed

Re: NIfi Registry duplicate key

Thank you Joe, didn't work either, cause registry 0.5 doesn't start when java v11. So tonight I will better upgrade nifi to 11.4 and hope that will work. Thanks, let you know. LC El lun, 18-05-2020 a las 14:58 -0400, Joe Witt escribió: > Luis, > I'm not saying they wont work together but Registry

Re: NIfi Registry duplicate key

This bug should be fixed in 1.11.0. There was a registry JIRA here [1] but it was really a fix on the NIFI side in NIFI-6872. [1] https://issues.apache.org/jira/browse/NIFIREG-347 [2] https://issues.apache.org/jira/browse/NIFI-6872 On Mon, May 18, 2020 at 2:59 PM Joe Witt wrote: > Luis, > >

Re: Nifi - how to achieve a concurrent development and CI/CD

Hi Eric, So I looked into this September last year. I found that a variant of master driven development was the best course of action, for collaboration on a single Nifi flow. As far as I can tell the git integration deliberately designed to be as simple a s possible, the down side of this is

Re: Nifi - how to achieve a concurrent development and CI/CD

Hi Ami, I'm testing our NiFi application using a blackbox approach. We have a test harness that stages input files and waits for the corresponding outputs, then validates the content of the output. NiFi runs in a docker container (see https://hub.docker.com/r/apache/nifi/) and you can automate

Re: Nifi - how to achieve a concurrent development and CI/CD

I think we could improve Nifi by hooking up to GitHub somehow. So that we don’t need the registry… > On 14. May 2020, at 18:49, Ami Goldenberg wrote: > > Hi Eric, > Would love to know, what kind of tests do you write/run for NiFi? We were > just researching this topic. > > On Thu, May 14,

Re: Nifi - how to achieve a concurrent development and CI/CD

Hi Eric, Would love to know, what kind of tests do you write/run for NiFi? We were just researching this topic. On Thu, May 14, 2020 at 6:38 PM Eric Secules wrote: > Hi Michal, > > I'm also using a single registry for development and production. It > doesn't help with collaborating on the same

Re: Nifi - how to achieve a concurrent development and CI/CD

Hi Michal, I'm also using a single registry for development and production. It doesn't help with collaborating on the same process group as there is way for it to reconcile merge conflicts. Instead, the registry will earn you that you're about to overwrite someone else's changes. Another pain of

Re: Nifi - how to achieve a concurrent development and CI/CD

Hi, Managing xml is always hard I think. Last time I need to do something similar we used https://nifi.apache.org/registry.html Works pretty well It was already 2 Years ago. Maybe now there is something better > On 14. May 2020, at 15:57, Michal Slama

Re: NiFi and real-time data lake

Yeah what a cool note and thanks for taking the time to share. On Tue, May 12, 2020 at 3:49 PM Andy LoPresto wrote: > Thanks Boris, this is really interesting to read and I appreciate that > you’re sharing it with the community. We’re glad NiFi can help with these > important use cases. > >

Re: NiFi and real-time data lake

Thanks Boris, this is really interesting to read and I appreciate that you’re sharing it with the community. We’re glad NiFi can help with these important use cases. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B

Re: NiFi Swagger/OpenAPI Processor

David, Fair enough. The attributes that it adds are documented. If you add the processor to your NiFi canvas and right-click on it and View Usage, you will get the processor’s documentation that indicates each of the attributes that are added along with an explanation of it. Or you could go

Re: NiFi Swagger/OpenAPI Processor

Thanks for the suggestions. I'm trying to find a way to handle this in a dynamic way based on the OpenAPI spec. Obviously I need to have a connected processor that can then generate the correct response for the request. You mentioned that that the HandleHttpRequest processor adds attributes

Re: NiFi Swagger/OpenAPI Processor

Dave, You may also want to consider using RouteOnAttribute. The HandleHttpRequest processor adds attributes for just about everything in the request. You can easily use those to route the requests not matching the path/method you want to a HandleHttpResponse that responds with a 404 or 405.

Re: NiFi Swagger/OpenAPI Processor

Maybe put an nginx in front that only passes through the requests you need? Or some other reverse proxy. I'm not affiliated with AWS but know their API gateway has this functionality https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-import-api.html On Tue, May 12, 2020,

Re: NiFi Swagger/OpenAPI Processor

If nobody has done this you can look at: https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-providingCodeOrDocumentationContributionProvidingcodeordocumentationcontributions On May 11, 2020 at

Re: NiFi Pubish Kafka with SASL OAUTHBEARER

Ok, great. Will be watching this closely :) Thanks Bryan Bende escreveu no dia segunda, 4/05/2020 à(s) 13:45: > Hello, > > Currently the only SASL mechanisms supported are GSSAPI, PLAIN, and SCRAM > SHA 256/512. > > In older NiFi releases (I think prior to 1.10), the SASL mechanism was not > a

Re: NiFi Pubish Kafka with SASL OAUTHBEARER

Hello, Currently the only SASL mechanisms supported are GSSAPI, PLAIN, and SCRAM SHA 256/512. In older NiFi releases (I think prior to 1.10), the SASL mechanism was not a first class property in the processor, so you could add a dynamic property and specify "sasl.mechanism" as whatever you want,

Re: NiFi Pubish Kafka with SASL OAUTHBEARER

Hello All, Does anyone know if there´s any way to publish kafka on NiFi using "SASL OAUTHBEARER"? https://docs.confluent.io/current/kafka/authentication_sasl/authentication_sasl_oauth.html Thanks in advance Bruno Gante escreveu no dia segunda, 4/05/2020 à(s) 12:39: > Hello All, > > Does

Re: NIFI - Connect processors properly

Hi Asmath, You can double click on the black line of a connection to add bending points. You can add as many as you want. You can also double click on a bending point to remove it. Hope this helps, Pierre Le mer. 29 avr. 2020 à 16:14, KhajaAsmath Mohammed a écrit : > Hi, > > How do I connect

Re: nifi 1.9.2 upgrade and zookeeper

great, thanks Drew! On Wed, Apr 22, 2020 at 1:37 PM Andrew Lim wrote: > Hi Boris, > > I think the information you are looking for is in the NiFi Migration > Guidance [1]. > > -Drew > > [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance > > On Apr 22, 2020, at 1:27 PM, Boris

<    1   2   3   4   5   6   7   8   9   10   >