RE: kafka-clients version
Yes: https://issues.apache.org/jira/browse/KAFKA-9815 > we'd have to be sure the latest 2.x client also works well with all previous > 2.x versions back to 2.0 I think kafka-clients should make sure of that. The processors could still be named 2_0, it’s only the client jar that should be the latest 2.x version. Specifically https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/pom.xml#L31 Regards Chris From: Bryan Bende Sent: Friday, 12. June 2020 15:18 To: users@nifi.apache.org Subject: Re: kafka-clients version Usually we make a new version of the processors when something changes that causes a previous version of the client to not work well with the newer broker (i.e. 0.9 client with 0.10 broker, in the older versions). The latest processors are named with "2_0" so not sure if it makes sense to upgrade the client lib and then the name is no longer accurate, plus we'd have to be sure the latest 2.x client also works well with all previous 2.x versions back to 2.0. The other option is to make a new copy of the processor like "2_5" or whatever the latest version is, but I think we only want to do this if it's really necessary. AFAIK the 2.0 client works with the latest 2.x brokers. Is there any specific issue you are looking to solve by using the latest client? On Fri, Jun 12, 2020 at 8:20 AM mailto:christophe.mon...@post.ch>> wrote: Hi Any reason why nifi-kafka-bundle is still bundled with kafka-clients-2.0.0.jar and not a newer version? Regards Chris
kafka-clients version
Hi Any reason why nifi-kafka-bundle is still bundled with kafka-clients-2.0.0.jar and not a newer version? Regards Chris
RE: Apache Nifi with IBM Event Streams
Hi It seems you want to configure ConsumeKafka / ConsumeKafkaRecord for SASL_SSL PLAIN. It is possible but not straightforward. The "Kerberos Service Name" cannot be empty but its content does not matter since it will not be used in this case. The trick is to specify the 2 additional properties sasl.jaas.config and sasl.mechanism, exactly like you did. You must create a Controller Service of type StandardSSLContextService, see https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-ssl-context-service-nar/1.5.0/org.apache.nifi.ssl.StandardSSLContextService/index.html And reference the Truststore so that Nifi can securely connect to your Kafka platform. Gruss Chris -Original Message- From: dkommineni Sent: Wednesday, October 9, 2019 21:35 To: users@nifi.apache.org Subject: Apache Nifi with IBM Event Streams I am trying to use Apache Nifi docker image "apache/nifi" to connect to Kafka in IBM Event Streams platform. I am trying to use "SASL_SSL" as the security protocol. My questions are: What is the "Kerberos Service Name" to be used? If some configuration required how and where to setup? What is the "SSL Context Service" and how do I create one? I tried adding additional properties like "sasl.mechanism=PLAIN"and "sasl.jaas.config" as org.apache.kafka.common.security.plain.PlainLoginModule required username="XxxxXXxxx password="XxxXxxxXXx"; -- Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
RE: NiFi 1.7.1 remote group not connecting when added through restful api until nifi restarted
The POST request that create an RPG returns an RemoteProcessGroupEntity whose ID can be used to call GET repeatedly. At some point you will find the port in RemoteProcessGroupEntity.getComponent().getContents().getInputPorts() but not immediately after creation. -Original Message- From: William Gosse Sent: Freitag, 16 November 2018 14:06 To: users@nifi.apache.org Subject: RE: NiFi 1.7.1 remote group not connecting when added through restful api until nifi restarted "My workaround is to repeatedly call GET on the RPG until the port becomes available." Can you explain a little more by what you mean by this. Or if you have an example please send it. -Original Message- From: christophe.mon...@post.ch Sent: Friday, November 16, 2018 12:46 AM To: users@nifi.apache.org Subject: RE: NiFi 1.7.1 remote group not connecting when added through restful api until nifi restarted [CAUTION: This email originated from outside of Kodak Alaris. Do not click links or open attachments unless you recognize the sender and know the content is safe.] Hi William My experience with using the REST API to create RPG is that you must wait “some time” before the port becomes available. A restart is not needed though. My workaround is to repeatedly call GET on the RPG until the port becomes available. It typically takes more than 10 seconds on a 5 nodes cluster. Chris -Original Message- From: William Gosse Sent: Donnerstag, 15 November 2018 18:03 To: users@nifi.apache.org Subject: RE: NiFi 1.7.1 remote group not connecting when added through restful api until nifi restarted I don't believe I see any error messages on the initial remote process group that gets created. I do see the error message when I do a restart and there are existing remote process groups but they do seem to connect eventually It's only the initial create of the first remote process group that seems to be acting weird. -Original Message- From: Koji Kawamura Sent: Thursday, November 15, 2018 1:05 AM To: users@nifi.apache.org Subject: Re: NiFi 1.7.1 remote group not connecting when added through restful api until nifi restarted [CAUTION: This email originated from outside of Kodak Alaris. Do not click links or open attachments unless you recognize the sender and know the content is safe.] Hello William, > fails to connect to the existing input port until I do a restart of > NiFi Is there any error message when it fails? Connection refused? It should not require a NiFi restart to establish connection. Thanks, Koji On Thu, Nov 15, 2018 at 1:38 AM William Gosse wrote: > > I'm using NiFi's restful api to create a process group from a template > that contains a file folder listener and a remote group. I also have > an existing input port going to a process group that has the file > fetch and everything else for processing the file that was fetched. > This is not added by a restful api but is loaded from a template after > the initial startup of Nifi. What I wind up with is a list fetch > pattern that should cluster ready/ > > My problem is that the first time I use the rest api to add the listener > process group the remote group that's inside of it fails to connect to the > existing input port until I do a restart of NiFi. At that point all > subsequent listeners I add seem to connect just fine. > > Is there way that I can cause the initial connection to occur without having > to perform a restart? > >
RE: NiFi 1.7.1 remote group not connecting when added through restful api until nifi restarted
Hi William My experience with using the REST API to create RPG is that you must wait “some time” before the port becomes available. A restart is not needed though. My workaround is to repeatedly call GET on the RPG until the port becomes available. It typically takes more than 10 seconds on a 5 nodes cluster. Chris -Original Message- From: William Gosse Sent: Donnerstag, 15 November 2018 18:03 To: users@nifi.apache.org Subject: RE: NiFi 1.7.1 remote group not connecting when added through restful api until nifi restarted I don't believe I see any error messages on the initial remote process group that gets created. I do see the error message when I do a restart and there are existing remote process groups but they do seem to connect eventually It's only the initial create of the first remote process group that seems to be acting weird. -Original Message- From: Koji Kawamura Sent: Thursday, November 15, 2018 1:05 AM To: users@nifi.apache.org Subject: Re: NiFi 1.7.1 remote group not connecting when added through restful api until nifi restarted [CAUTION: This email originated from outside of Kodak Alaris. Do not click links or open attachments unless you recognize the sender and know the content is safe.] Hello William, > fails to connect to the existing input port until I do a restart of > NiFi Is there any error message when it fails? Connection refused? It should not require a NiFi restart to establish connection. Thanks, Koji On Thu, Nov 15, 2018 at 1:38 AM William Gosse wrote: > > I'm using NiFi's restful api to create a process group from a template > that contains a file folder listener and a remote group. I also have > an existing input port going to a process group that has the file > fetch and everything else for processing the file that was fetched. > This is not added by a restful api but is loaded from a template after > the initial startup of Nifi. What I wind up with is a list fetch > pattern that should cluster ready/ > > My problem is that the first time I use the rest api to add the listener > process group the remote group that's inside of it fails to connect to the > existing input port until I do a restart of NiFi. At that point all > subsequent listeners I add seem to connect just fine. > > Is there way that I can cause the initial connection to occur without having > to perform a restart? > >
RE: NiFi 1.7.1 remote group not connecting when added through restful api until nifi restarted
Hi William My experience with using the REST API to create RPG is that you must wait "some time" before the port becomes available. A restart is not needed though. My workaround is to repeatedly call GET on the RPG until the port becomes available. It typically takes more than 10 seconds on a 5 nodes cluster. Chris From: William Gosse Sent: Mittwoch, 14 November 2018 17:39 To: users@nifi.apache.org Subject: NiFi 1.7.1 remote group not connecting when added through restful api until nifi restarted I'm using NiFi's restful api to create a process group from a template that contains a file folder listener and a remote group. I also have an existing input port going to a process group that has the file fetch and everything else for processing the file that was fetched. This is not added by a restful api but is loaded from a template after the initial startup of Nifi. What I wind up with is a list fetch pattern that should cluster ready/ My problem is that the first time I use the rest api to add the listener process group the remote group that's inside of it fails to connect to the existing input port until I do a restart of NiFi. At that point all subsequent listeners I add seem to connect just fine. Is there way that I can cause the initial connection to occur without having to perform a restart?
ExecuteSQL Avro schema: all fields are nullable
Hello Is there a reason why JdbcCommon#createSchema creates an Avro schema with nullable types for all fields? Why not check with java.sql.ResultSetMetaData#isNullable ? https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.1/org.apache.nifi.processors.standard.ExecuteSQL/index.html https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java#L500 Regards, Chris
