Oleg, It's part of the Cloudera distro. Not sure of the lineage beyond that. Here are a couple of links.
https://community.cloudera.com/t5/Data-Ingestion-Integration/New-Kafka-0-8-2-0-1-kafka1-3-1-p0-9-Parcel-What-are-the-changes/td-p/30506 http://archive.cloudera.com/kafka/parcels/1.3.1/ Michael On Tue, Mar 15, 2016 at 12:01 PM, Oleg Zhurakousky < [email protected]> wrote: > Michael > > What is KAFKA-0.8.2.0-1.kafka1.3.1.p0.9? I mean where can I get that > build? > I guess based on the previous email we’ve tested our code with 3 versions > of ASF distribution of Kafka and the above version tells me that it may be > some kind of fork. > > Also, we are considering downgrading Kafka dependencies back to the 0.8 > and of 0.7 provide a new version of Kafka processors that utilize Kafka 0.9 > new producer/consumer API > > Thanks > Oleg > > On Mar 15, 2016, at 11:46 AM, Michael Dyer <[email protected]> > wrote: > > Joe, > > I'm seeing a similar issue moving from 0.3.0 to 0.5.1 with > KAFKA-0.8.2.0-1.kafka1.3.1.p0.9. > > I can see the tasks/time counter increment on the processor but no flow > data ever leaves the processor. There are no errors shown in the bulletin > board. The app log shows below (repeating). > > The rename 0.4.1 nar to 0.5.1 nar trick (restart nifi) works, except that > the '*batch size*' value does not seem to be honored. I have my batch > size set to 10000, but I'm seeing files written continually (every few > seconds) with much smaller sizes. I suspect this has to do with the ` > *auto.offset.reset`* value which defaults to `*largest*`. From what I > have read 'smallest' causes the client to start at the beginning which > sounds like I would be retrieving duplicates. > > Renaming 0.3.0 nar to 0.5.1 (restart nifi) restores the original behavior. > > yBuffer([[netflow5,0], initOffset 297426 to broker BrokerEndPoint(176, > n2.foo.bar.com,9092)] ) > 2016-03-15 07:45:17,237 WARN > [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176] > kafka.consumer.ConsumerFetcherThread > [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], > Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@11bd00f6. > Possible cause: java.lang.IllegalArgumentException > 2016-03-15 07:45:17,443 INFO > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] > kafka.utils.VerifiableProperties Verifying properties > 2016-03-15 07:45:17,443 INFO > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] > kafka.utils.VerifiableProperties Property client.id is overridden to > NiFi-b6c67ee3-aa9e-419d-8a57-84ab5e76c017 > 2016-03-15 07:45:17,443 INFO > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] > kafka.utils.VerifiableProperties Property metadata.broker.list is > overridden to n3.foo.bar.com:9092,n2.foo.bar.com:9092,n4.foo.bar.com:9092 > 2016-03-15 07:45:17,443 INFO > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] > kafka.utils.VerifiableProperties Property request.timeout.ms is > overridden to 30000 > 2016-03-15 07:45:17,443 INFO > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] > kafka.client.ClientUtils$ Fetching metadata from broker BrokerEndPoint(196, > n4.foo.bar.com,9092) with correlation id 14 for 1 topic(s) Set(netflow5) > 2016-03-15 07:45:17,443 INFO > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] > kafka.producer.SyncProducer Connected to n4.foo.bar.com:9092 for producing > 2016-03-15 07:45:17,444 INFO > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] > kafka.producer.SyncProducer Disconnecting from n4.foo.bar.com:9092 > 2016-03-15 07:45:17,444 INFO > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] > kafka.consumer.ConsumerFetcherManager > [ConsumerFetcherManager-1458053114395] Added fetcher for partitions > ArrayBuffer([[netflow5,0], initOffset 297426 to broker BrokerEndPoint(176, > n2.foo.bar.com,9092)] ) > 2016-03-15 07:45:17,449 WARN > [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176] > kafka.consumer.ConsumerFetcherThread > [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], > Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@3e69ef73. > Possible cause: java.lang.IllegalArgumentException > 2016-03-15 07:45:17,626 INFO [NiFi Web Server-259] > o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling > GetKafka[id=4943a24e-af5c-4392-bc45-7008f30674bb] to run > 2016-03-15 07:45:17,626 INFO [Timer-Driven Process Thread-3] > k.consumer.ZookeeperConsumerConnector > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0], > ZKConsumerConnector shutting down > 2016-03-15 07:45:17,632 INFO [Timer-Driven Process Thread-3] > kafka.consumer.ConsumerFetcherManager > [ConsumerFetcherManager-1458053114395] Stopping leader finder thread > 2016-03-15 07:45:17,633 INFO [Timer-Driven Process Thread-3] > k.c.ConsumerFetcherManager$LeaderFinderThread > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread], > Shutting down > 2016-03-15 07:45:17,634 INFO > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] > k.c.ConsumerFetcherManager$LeaderFinderThread > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread], > Stopped > 2016-03-15 07:45:17,634 INFO [Timer-Driven Process Thread-3] > k.c.ConsumerFetcherManager$LeaderFinderThread > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread], > Shutdown completed > 2016-03-15 07:45:17,634 INFO [Timer-Driven Process Thread-3] > kafka.consumer.ConsumerFetcherManager > [ConsumerFetcherManager-1458053114395] Stopping all fetchers > 2016-03-15 07:45:17,634 INFO [Timer-Driven Process Thread-3] > kafka.consumer.ConsumerFetcherThread > [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], > Shutting down > 2016-03-15 07:45:17,634 INFO > [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176] > kafka.consumer.ConsumerFetcherThread > [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], > Stopped > 2016-03-15 07:45:17,635 INFO [Timer-Driven Process Thread-3] > kafka.consumer.ConsumerFetcherThread > [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], > Shutdown completed > 2016-03-15 07:45:17,635 INFO [Timer-Driven Process Thread-3] > kafka.consumer.ConsumerFetcherManager > [ConsumerFetcherManager-1458053114395] All connections stopped > 2016-03-15 07:45:17,635 INFO [ZkClient-EventThread-302-192.168.1.1:2181] > org.I0Itec.zkclient.ZkEventThread Terminate ZkClient event thread. > 2016-03-15 07:45:17,638 INFO [Timer-Driven Process Thread-3] > org.apache.zookeeper.ZooKeeper Session: 0x1535e2aa53b3f61 closed > 2016-03-15 07:45:17,638 INFO [Timer-Driven Process Thread-3] > k.consumer.ZookeeperConsumerConnector > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0], > ZKConsumerConnector shutdown completed in 11 ms > 2016-03-15 07:45:17,639 INFO [Timer-Driven Process Thread-4-EventThread] > org.apache.zookeeper.ClientCnxn EventThread shut down > 2016-03-15 07:45:17,745 INFO [Flow Service Tasks Thread-2] > o.a.nifi.controller.StandardFlowService Saved flow controller > org.apache.nifi.controller.FlowController@45d22bd5 // Another save > pending = false > 2016-03-15 07:45:18,414 INFO > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0_watcher_executor] > k.consumer.ZookeeperConsumerConnector > [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0], > stopping watcher executor thread for consumer > b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0 > > Hope this helps... > > Michael > > On Mon, Feb 22, 2016 at 4:12 PM, Joe Witt <[email protected]> wrote: > >> Sorry to clarify it working against all three of these at once: >> - Kafka 0.8.1.1 >> - Kafka 0.8.2.1 >> - Kafka 0.9.0.1 >> >> Thanks >> Joe >> >> On Mon, Feb 22, 2016 at 4:12 PM, Joe Witt <[email protected]> wrote: >> > All just as a point of reference we now have a live system running >> > that is on NiFi 0.5.0 and feeding three versions of Kafka at once: >> > - 0.8.1 >> > - 0.8.2.0 >> > - 0.9.0.1 >> > >> > So perhaps there are some particular configurations that cause issues. >> > Can you share more details about your configuration of Kafka/NiFi and >> > what sort of security is enabled? >> > >> > Thanks >> > Joe >> > >> > On Mon, Feb 22, 2016 at 1:01 PM, Kyle Burke <[email protected]> >> wrote: >> >> I replaced my 0.5.0 kafka nar with the 0.4.1 kakfa nar and it fixed my >> kafka >> >> issue. I renamed the 0.4.1 nar to be 0.5.0.nar and restart nifi and my >> kafka >> >> processor started reading my 0.8.2.1 stream. Not elegant but glad it >> worked. >> >> >> >> >> >> Respectfully, >> >> >> >> Kyle Burke | Data Science Engineer >> >> IgnitionOne - Marketing Technology. Simplified. >> >> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 >> >> Direct: 404.961.3918 >> >> >> >> >> >> From: Joe Witt >> >> Reply-To: "[email protected]" >> >> Date: Sunday, February 21, 2016 at 5:23 PM >> >> To: "[email protected]" >> >> Subject: Re: Nifi 0.50 and GetKafka Issues >> >> >> >> Yeah the intent is to support 0.8 and 0.9. Will figure something out. >> >> >> >> Thanks >> >> Joe >> >> >> >> On Feb 21, 2016 4:47 PM, "West, Joshua" <[email protected]> wrote: >> >>> >> >>> Hi Oleg, >> >>> >> >>> Hmm -- from what I can tell, this isn't a Zookeeper communication >> issue. >> >>> Nifi is able to connect into the Kafka brokers' Zookeeper cluster and >> >>> retrieve the list of the kafka brokers to connect to. Seems, from >> the logs, >> >>> to be a problem when attempting to consume from Kafka itself. >> >>> >> >>> I'm guessing that the Kafka 0.9.0 client libraries just are not >> compatible >> >>> with Kafka 0.8.2.1 so in order to use Nifi 0.5.0 with Kafka, the Kafka >> >>> version must be >= 0.9.0. >> >>> >> >>> Any change Nifi could add backwards compatible support for Kafka >> 0.8.2.1 >> >>> too? Let you choose which client library version, when setting up the >> >>> GetKafka processor? >> >>> >> >>> -- >> >>> Josh West <[email protected]> >> >>> Bose Corporation >> >>> >> >>> >> >>> On Sun, 2016-02-21 at 15:02 +0000, Oleg Zhurakousky wrote: >> >>> >> >>> Josh >> >>> >> >>> Also, keep in mind that there are incompatible property names in Kafka >> >>> between the 0.7 and 0.8 releases. One of the change that went it was >> >>> replacing “zk.connectiontimeout.ms” with “ >> zookeeper.connection.timeout.ms”. >> >>> Not sure if it’s related though, but realizing that 0.4.1 was relying >> on >> >>> this property it’s value was completely ignored with 0.8 client >> libraries >> >>> (you could actually see the WARN message to that effect) and now it >> is not >> >>> ignored, so take a look and see if tinkering with its value changes >> >>> something. >> >>> >> >>> Cheers >> >>> Oleg >> >>> >> >>> On Feb 20, 2016, at 6:47 PM, Oleg Zhurakousky >> >>> <[email protected]> wrote: >> >>> >> >>> Josh >> >>> >> >>> The only change that ’s went and relevant to your issue is the fact >> that >> >>> we’ve upgraded client libraries to Kafka 0.9 and between 0.8 and 0.9 >> Kafka >> >>> introduced wire protocol changes that break compatibility. >> >>> I am still digging so stay tuned. >> >>> >> >>> Oleg >> >>> >> >>> On Feb 20, 2016, at 4:10 PM, West, Joshua <[email protected]> wrote: >> >>> >> >>> Hi Oleg and Joe, >> >>> >> >>> Kafka 0.8.2.1 >> >>> >> >>> Attached is the app log with hostnames scrubbed. >> >>> >> >>> Thanks for your help. Much appreciated. >> >>> >> >>> -- >> >>> Josh West <[email protected]> >> >>> Bose Corporation >> >>> >> >>> >> >>> On Sat, 2016-02-20 at 15:46 -0500, Joe Witt wrote: >> >>> >> >>> And also what version of Kafka are you using? >> >>> >> >>> On Feb 20, 2016 3:37 PM, "Oleg Zhurakousky" < >> [email protected]> >> >>> wrote: >> >>> >> >>> Josh >> >>> >> >>> Any chance to attache the app-log or relevant stack trace? >> >>> >> >>> Thanks >> >>> Oleg >> >>> >> >>> On Feb 20, 2016, at 3:30 PM, West, Joshua <[email protected]> wrote: >> >>> >> >>> Hi folks, >> >>> >> >>> I've upgraded from Nifi 0.4.1 to 0.5.0 and I am no longer able to use >> the >> >>> GetKafka processor. I'm seeing errors like so: >> >>> >> >>> 2016-02-20 20:10:14,953 WARN >> >>> >> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0] >> >>> kafka.consumer.ConsumerFetcherThread >> >>> >> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0], >> >>> Error in >> fetchkafka.consumer.ConsumerFetcherThread$FetchRequest@7b49a642. >> >>> Possible cause: java.lang.IllegalArgumentException >> >>> >> >>> ^ Note the hostname of the server has been scrubbed. >> >>> >> >>> My configuration is pretty generic, except that with Zookeeper we use >> a >> >>> different root path, so our Zookeeper connect string looks like so: >> >>> >> >>> zookeeper-node1:2181,zookeeper-node2:2181,zookeeper-node3:2181/kafka >> >>> >> >>> Is anybody else experiencing issues? >> >>> >> >>> Thanks. >> >>> >> >>> -- >> >>> Josh West <[email protected]> >> >>> >> >>> Cloud Architect >> >>> Bose Corporation >> >>> >> >>> >> >>> >> >>> >> >>> <nifi-app.log.kafkaissues.bz2> >> >>> >> >>> >> >>> >> >> >> > > >
