All just as a point of reference we now have a live system running that is on NiFi 0.5.0 and feeding three versions of Kafka at once: - 0.8.1 - 0.8.2.0 - 0.9.0.1
So perhaps there are some particular configurations that cause issues. Can you share more details about your configuration of Kafka/NiFi and what sort of security is enabled? Thanks Joe On Mon, Feb 22, 2016 at 1:01 PM, Kyle Burke <kyle.bu...@ignitionone.com> wrote: > I replaced my 0.5.0 kafka nar with the 0.4.1 kakfa nar and it fixed my kafka > issue. I renamed the 0.4.1 nar to be 0.5.0.nar and restart nifi and my kafka > processor started reading my 0.8.2.1 stream. Not elegant but glad it worked. > > > Respectfully, > > Kyle Burke | Data Science Engineer > IgnitionOne - Marketing Technology. Simplified. > Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 > Direct: 404.961.3918 > > > From: Joe Witt > Reply-To: "users@nifi.apache.org" > Date: Sunday, February 21, 2016 at 5:23 PM > To: "users@nifi.apache.org" > Subject: Re: Nifi 0.50 and GetKafka Issues > > Yeah the intent is to support 0.8 and 0.9. Will figure something out. > > Thanks > Joe > > On Feb 21, 2016 4:47 PM, "West, Joshua" <josh_w...@bose.com> wrote: >> >> Hi Oleg, >> >> Hmm -- from what I can tell, this isn't a Zookeeper communication issue. >> Nifi is able to connect into the Kafka brokers' Zookeeper cluster and >> retrieve the list of the kafka brokers to connect to. Seems, from the logs, >> to be a problem when attempting to consume from Kafka itself. >> >> I'm guessing that the Kafka 0.9.0 client libraries just are not compatible >> with Kafka 0.8.2.1 so in order to use Nifi 0.5.0 with Kafka, the Kafka >> version must be >= 0.9.0. >> >> Any change Nifi could add backwards compatible support for Kafka 0.8.2.1 >> too? Let you choose which client library version, when setting up the >> GetKafka processor? >> >> -- >> Josh West <josh_w...@bose.com> >> Bose Corporation >> >> >> On Sun, 2016-02-21 at 15:02 +0000, Oleg Zhurakousky wrote: >> >> Josh >> >> Also, keep in mind that there are incompatible property names in Kafka >> between the 0.7 and 0.8 releases. One of the change that went it was >> replacing “zk.connectiontimeout.ms” with “zookeeper.connection.timeout.ms”. >> Not sure if it’s related though, but realizing that 0.4.1 was relying on >> this property it’s value was completely ignored with 0.8 client libraries >> (you could actually see the WARN message to that effect) and now it is not >> ignored, so take a look and see if tinkering with its value changes >> something. >> >> Cheers >> Oleg >> >> On Feb 20, 2016, at 6:47 PM, Oleg Zhurakousky >> <ozhurakou...@hortonworks.com> wrote: >> >> Josh >> >> The only change that ’s went and relevant to your issue is the fact that >> we’ve upgraded client libraries to Kafka 0.9 and between 0.8 and 0.9 Kafka >> introduced wire protocol changes that break compatibility. >> I am still digging so stay tuned. >> >> Oleg >> >> On Feb 20, 2016, at 4:10 PM, West, Joshua <josh_w...@bose.com> wrote: >> >> Hi Oleg and Joe, >> >> Kafka 0.8.2.1 >> >> Attached is the app log with hostnames scrubbed. >> >> Thanks for your help. Much appreciated. >> >> -- >> Josh West <josh_w...@bose.com> >> Bose Corporation >> >> >> On Sat, 2016-02-20 at 15:46 -0500, Joe Witt wrote: >> >> And also what version of Kafka are you using? >> >> On Feb 20, 2016 3:37 PM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> >> wrote: >> >> Josh >> >> Any chance to attache the app-log or relevant stack trace? >> >> Thanks >> Oleg >> >> On Feb 20, 2016, at 3:30 PM, West, Joshua <josh_w...@bose.com> wrote: >> >> Hi folks, >> >> I've upgraded from Nifi 0.4.1 to 0.5.0 and I am no longer able to use the >> GetKafka processor. I'm seeing errors like so: >> >> 2016-02-20 20:10:14,953 WARN >> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0] >> kafka.consumer.ConsumerFetcherThread >> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0], >> Error in fetchkafka.consumer.ConsumerFetcherThread$FetchRequest@7b49a642. >> Possible cause: java.lang.IllegalArgumentException >> >> ^ Note the hostname of the server has been scrubbed. >> >> My configuration is pretty generic, except that with Zookeeper we use a >> different root path, so our Zookeeper connect string looks like so: >> >> zookeeper-node1:2181,zookeeper-node2:2181,zookeeper-node3:2181/kafka >> >> Is anybody else experiencing issues? >> >> Thanks. >> >> -- >> Josh West <josh_w...@bose.com> >> >> Cloud Architect >> Bose Corporation >> >> >> >> >> <nifi-app.log.kafkaissues.bz2> >> >> >> >
