Sorry to clarify it working against all three of these at once: - Kafka 0.8.1.1 - Kafka 0.8.2.1 - Kafka 0.9.0.1
Thanks Joe On Mon, Feb 22, 2016 at 4:12 PM, Joe Witt <joe.w...@gmail.com> wrote: > All just as a point of reference we now have a live system running > that is on NiFi 0.5.0 and feeding three versions of Kafka at once: > - 0.8.1 > - 0.8.2.0 > - 0.9.0.1 > > So perhaps there are some particular configurations that cause issues. > Can you share more details about your configuration of Kafka/NiFi and > what sort of security is enabled? > > Thanks > Joe > > On Mon, Feb 22, 2016 at 1:01 PM, Kyle Burke <kyle.bu...@ignitionone.com> > wrote: >> I replaced my 0.5.0 kafka nar with the 0.4.1 kakfa nar and it fixed my kafka >> issue. I renamed the 0.4.1 nar to be 0.5.0.nar and restart nifi and my kafka >> processor started reading my 0.8.2.1 stream. Not elegant but glad it worked. >> >> >> Respectfully, >> >> Kyle Burke | Data Science Engineer >> IgnitionOne - Marketing Technology. Simplified. >> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 >> Direct: 404.961.3918 >> >> >> From: Joe Witt >> Reply-To: "users@nifi.apache.org" >> Date: Sunday, February 21, 2016 at 5:23 PM >> To: "users@nifi.apache.org" >> Subject: Re: Nifi 0.50 and GetKafka Issues >> >> Yeah the intent is to support 0.8 and 0.9. Will figure something out. >> >> Thanks >> Joe >> >> On Feb 21, 2016 4:47 PM, "West, Joshua" <josh_w...@bose.com> wrote: >>> >>> Hi Oleg, >>> >>> Hmm -- from what I can tell, this isn't a Zookeeper communication issue. >>> Nifi is able to connect into the Kafka brokers' Zookeeper cluster and >>> retrieve the list of the kafka brokers to connect to. Seems, from the logs, >>> to be a problem when attempting to consume from Kafka itself. >>> >>> I'm guessing that the Kafka 0.9.0 client libraries just are not compatible >>> with Kafka 0.8.2.1 so in order to use Nifi 0.5.0 with Kafka, the Kafka >>> version must be >= 0.9.0. >>> >>> Any change Nifi could add backwards compatible support for Kafka 0.8.2.1 >>> too? Let you choose which client library version, when setting up the >>> GetKafka processor? >>> >>> -- >>> Josh West <josh_w...@bose.com> >>> Bose Corporation >>> >>> >>> On Sun, 2016-02-21 at 15:02 +0000, Oleg Zhurakousky wrote: >>> >>> Josh >>> >>> Also, keep in mind that there are incompatible property names in Kafka >>> between the 0.7 and 0.8 releases. One of the change that went it was >>> replacing “zk.connectiontimeout.ms” with “zookeeper.connection.timeout.ms”. >>> Not sure if it’s related though, but realizing that 0.4.1 was relying on >>> this property it’s value was completely ignored with 0.8 client libraries >>> (you could actually see the WARN message to that effect) and now it is not >>> ignored, so take a look and see if tinkering with its value changes >>> something. >>> >>> Cheers >>> Oleg >>> >>> On Feb 20, 2016, at 6:47 PM, Oleg Zhurakousky >>> <ozhurakou...@hortonworks.com> wrote: >>> >>> Josh >>> >>> The only change that ’s went and relevant to your issue is the fact that >>> we’ve upgraded client libraries to Kafka 0.9 and between 0.8 and 0.9 Kafka >>> introduced wire protocol changes that break compatibility. >>> I am still digging so stay tuned. >>> >>> Oleg >>> >>> On Feb 20, 2016, at 4:10 PM, West, Joshua <josh_w...@bose.com> wrote: >>> >>> Hi Oleg and Joe, >>> >>> Kafka 0.8.2.1 >>> >>> Attached is the app log with hostnames scrubbed. >>> >>> Thanks for your help. Much appreciated. >>> >>> -- >>> Josh West <josh_w...@bose.com> >>> Bose Corporation >>> >>> >>> On Sat, 2016-02-20 at 15:46 -0500, Joe Witt wrote: >>> >>> And also what version of Kafka are you using? >>> >>> On Feb 20, 2016 3:37 PM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> >>> wrote: >>> >>> Josh >>> >>> Any chance to attache the app-log or relevant stack trace? >>> >>> Thanks >>> Oleg >>> >>> On Feb 20, 2016, at 3:30 PM, West, Joshua <josh_w...@bose.com> wrote: >>> >>> Hi folks, >>> >>> I've upgraded from Nifi 0.4.1 to 0.5.0 and I am no longer able to use the >>> GetKafka processor. I'm seeing errors like so: >>> >>> 2016-02-20 20:10:14,953 WARN >>> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0] >>> kafka.consumer.ConsumerFetcherThread >>> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0], >>> Error in fetchkafka.consumer.ConsumerFetcherThread$FetchRequest@7b49a642. >>> Possible cause: java.lang.IllegalArgumentException >>> >>> ^ Note the hostname of the server has been scrubbed. >>> >>> My configuration is pretty generic, except that with Zookeeper we use a >>> different root path, so our Zookeeper connect string looks like so: >>> >>> zookeeper-node1:2181,zookeeper-node2:2181,zookeeper-node3:2181/kafka >>> >>> Is anybody else experiencing issues? >>> >>> Thanks. >>> >>> -- >>> Josh West <josh_w...@bose.com> >>> >>> Cloud Architect >>> Bose Corporation >>> >>> >>> >>> >>> <nifi-app.log.kafkaissues.bz2> >>> >>> >>> >>
