Michael What is KAFKA-0.8.2.0-1.kafka1.3.1.p0.9? I mean where can I get that build? I guess based on the previous email we’ve tested our code with 3 versions of ASF distribution of Kafka and the above version tells me that it may be some kind of fork.
Also, we are considering downgrading Kafka dependencies back to the 0.8 and of 0.7 provide a new version of Kafka processors that utilize Kafka 0.9 new producer/consumer API Thanks Oleg On Mar 15, 2016, at 11:46 AM, Michael Dyer <[email protected]<mailto:[email protected]>> wrote: Joe, I'm seeing a similar issue moving from 0.3.0 to 0.5.1 with KAFKA-0.8.2.0-1.kafka1.3.1.p0.9. I can see the tasks/time counter increment on the processor but no flow data ever leaves the processor. There are no errors shown in the bulletin board. The app log shows below (repeating). The rename 0.4.1 nar to 0.5.1 nar trick (restart nifi) works, except that the 'batch size' value does not seem to be honored. I have my batch size set to 10000, but I'm seeing files written continually (every few seconds) with much smaller sizes. I suspect this has to do with the `auto.offset.reset` value which defaults to `largest`. From what I have read 'smallest' causes the client to start at the beginning which sounds like I would be retrieving duplicates. Renaming 0.3.0 nar to 0.5.1 (restart nifi) restores the original behavior. yBuffer([[netflow5,0], initOffset 297426 to broker BrokerEndPoint(176,n2.foo.bar.com<http://n2.foo.bar.com/>,9092)] ) 2016-03-15 07:45:17,237 WARN [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176] kafka.consumer.ConsumerFetcherThread [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@11bd00f6. Possible cause: java.lang.IllegalArgumentException 2016-03-15 07:45:17,443 INFO [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] kafka.utils.VerifiableProperties Verifying properties 2016-03-15 07:45:17,443 INFO [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] kafka.utils.VerifiableProperties Property client.id<http://client.id/> is overridden to NiFi-b6c67ee3-aa9e-419d-8a57-84ab5e76c017 2016-03-15 07:45:17,443 INFO [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] kafka.utils.VerifiableProperties Property metadata.broker.list is overridden to n3.foo.bar.com:9092<http://n3.foo.bar.com:9092/>,n2.foo.bar.com:9092<http://n2.foo.bar.com:9092/>,n4.foo.bar.com:9092<http://n4.foo.bar.com:9092/> 2016-03-15 07:45:17,443 INFO [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] kafka.utils.VerifiableProperties Property request.timeout.ms<http://request.timeout.ms/> is overridden to 30000 2016-03-15 07:45:17,443 INFO [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] kafka.client.ClientUtils$ Fetching metadata from broker BrokerEndPoint(196,n4.foo.bar.com<http://n4.foo.bar.com/>,9092) with correlation id 14 for 1 topic(s) Set(netflow5) 2016-03-15 07:45:17,443 INFO [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] kafka.producer.SyncProducer Connected to n4.foo.bar.com:9092<http://n4.foo.bar.com:9092/> for producing 2016-03-15 07:45:17,444 INFO [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] kafka.producer.SyncProducer Disconnecting from n4.foo.bar.com:9092<http://n4.foo.bar.com:9092/> 2016-03-15 07:45:17,444 INFO [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1458053114395] Added fetcher for partitions ArrayBuffer([[netflow5,0], initOffset 297426 to broker BrokerEndPoint(176,n2.foo.bar.com<http://n2.foo.bar.com/>,9092)] ) 2016-03-15 07:45:17,449 WARN [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176] kafka.consumer.ConsumerFetcherThread [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@3e69ef73. Possible cause: java.lang.IllegalArgumentException 2016-03-15 07:45:17,626 INFO [NiFi Web Server-259] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling GetKafka[id=4943a24e-af5c-4392-bc45-7008f30674bb] to run 2016-03-15 07:45:17,626 INFO [Timer-Driven Process Thread-3] k.consumer.ZookeeperConsumerConnector [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0], ZKConsumerConnector shutting down 2016-03-15 07:45:17,632 INFO [Timer-Driven Process Thread-3] kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1458053114395] Stopping leader finder thread 2016-03-15 07:45:17,633 INFO [Timer-Driven Process Thread-3] k.c.ConsumerFetcherManager$LeaderFinderThread [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread], Shutting down 2016-03-15 07:45:17,634 INFO [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] k.c.ConsumerFetcherManager$LeaderFinderThread [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread], Stopped 2016-03-15 07:45:17,634 INFO [Timer-Driven Process Thread-3] k.c.ConsumerFetcherManager$LeaderFinderThread [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread], Shutdown completed 2016-03-15 07:45:17,634 INFO [Timer-Driven Process Thread-3] kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1458053114395] Stopping all fetchers 2016-03-15 07:45:17,634 INFO [Timer-Driven Process Thread-3] kafka.consumer.ConsumerFetcherThread [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], Shutting down 2016-03-15 07:45:17,634 INFO [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176] kafka.consumer.ConsumerFetcherThread [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], Stopped 2016-03-15 07:45:17,635 INFO [Timer-Driven Process Thread-3] kafka.consumer.ConsumerFetcherThread [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], Shutdown completed 2016-03-15 07:45:17,635 INFO [Timer-Driven Process Thread-3] kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1458053114395] All connections stopped 2016-03-15 07:45:17,635 INFO [ZkClient-EventThread-302-192.168.1.1:2181] org.I0Itec.zkclient.ZkEventThread Terminate ZkClient event thread. 2016-03-15 07:45:17,638 INFO [Timer-Driven Process Thread-3] org.apache.zookeeper.ZooKeeper Session: 0x1535e2aa53b3f61 closed 2016-03-15 07:45:17,638 INFO [Timer-Driven Process Thread-3] k.consumer.ZookeeperConsumerConnector [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0], ZKConsumerConnector shutdown completed in 11 ms 2016-03-15 07:45:17,639 INFO [Timer-Driven Process Thread-4-EventThread] org.apache.zookeeper.ClientCnxn EventThread shut down 2016-03-15 07:45:17,745 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@45d22bd5 // Another save pending = false 2016-03-15 07:45:18,414 INFO [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0_watcher_executor] k.consumer.ZookeeperConsumerConnector [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0], stopping watcher executor thread for consumer b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0 Hope this helps... Michael On Mon, Feb 22, 2016 at 4:12 PM, Joe Witt <[email protected]<mailto:[email protected]>> wrote: Sorry to clarify it working against all three of these at once: - Kafka 0.8.1.1 - Kafka 0.8.2.1 - Kafka 0.9.0.1 Thanks Joe On Mon, Feb 22, 2016 at 4:12 PM, Joe Witt <[email protected]<mailto:[email protected]>> wrote: > All just as a point of reference we now have a live system running > that is on NiFi 0.5.0 and feeding three versions of Kafka at once: > - 0.8.1 > - 0.8.2.0 > - 0.9.0.1 > > So perhaps there are some particular configurations that cause issues. > Can you share more details about your configuration of Kafka/NiFi and > what sort of security is enabled? > > Thanks > Joe > > On Mon, Feb 22, 2016 at 1:01 PM, Kyle Burke > <[email protected]<mailto:[email protected]>> wrote: >> I replaced my 0.5.0 kafka nar with the 0.4.1 kakfa nar and it fixed my kafka >> issue. I renamed the 0.4.1 nar to be 0.5.0.nar and restart nifi and my kafka >> processor started reading my 0.8.2.1 stream. Not elegant but glad it worked. >> >> >> Respectfully, >> >> Kyle Burke | Data Science Engineer >> IgnitionOne - Marketing Technology. Simplified. >> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 >> Direct: 404.961.3918 >> >> >> From: Joe Witt >> Reply-To: "[email protected]<mailto:[email protected]>" >> Date: Sunday, February 21, 2016 at 5:23 PM >> To: "[email protected]<mailto:[email protected]>" >> Subject: Re: Nifi 0.50 and GetKafka Issues >> >> Yeah the intent is to support 0.8 and 0.9. Will figure something out. >> >> Thanks >> Joe >> >> On Feb 21, 2016 4:47 PM, "West, Joshua" >> <[email protected]<mailto:[email protected]>> wrote: >>> >>> Hi Oleg, >>> >>> Hmm -- from what I can tell, this isn't a Zookeeper communication issue. >>> Nifi is able to connect into the Kafka brokers' Zookeeper cluster and >>> retrieve the list of the kafka brokers to connect to. Seems, from the logs, >>> to be a problem when attempting to consume from Kafka itself. >>> >>> I'm guessing that the Kafka 0.9.0 client libraries just are not compatible >>> with Kafka 0.8.2.1 so in order to use Nifi 0.5.0 with Kafka, the Kafka >>> version must be >= 0.9.0. >>> >>> Any change Nifi could add backwards compatible support for Kafka 0.8.2.1 >>> too? Let you choose which client library version, when setting up the >>> GetKafka processor? >>> >>> -- >>> Josh West <[email protected]<mailto:[email protected]>> >>> Bose Corporation >>> >>> >>> On Sun, 2016-02-21 at 15:02 +0000, Oleg Zhurakousky wrote: >>> >>> Josh >>> >>> Also, keep in mind that there are incompatible property names in Kafka >>> between the 0.7 and 0.8 releases. One of the change that went it was >>> replacing “zk.connectiontimeout.ms<http://zk.connectiontimeout.ms/>” with >>> “zookeeper.connection.timeout.ms<http://zookeeper.connection.timeout.ms/>”. >>> Not sure if it’s related though, but realizing that 0.4.1 was relying on >>> this property it’s value was completely ignored with 0.8 client libraries >>> (you could actually see the WARN message to that effect) and now it is not >>> ignored, so take a look and see if tinkering with its value changes >>> something. >>> >>> Cheers >>> Oleg >>> >>> On Feb 20, 2016, at 6:47 PM, Oleg Zhurakousky >>> <[email protected]<mailto:[email protected]>> wrote: >>> >>> Josh >>> >>> The only change that ’s went and relevant to your issue is the fact that >>> we’ve upgraded client libraries to Kafka 0.9 and between 0.8 and 0.9 Kafka >>> introduced wire protocol changes that break compatibility. >>> I am still digging so stay tuned. >>> >>> Oleg >>> >>> On Feb 20, 2016, at 4:10 PM, West, Joshua >>> <[email protected]<mailto:[email protected]>> wrote: >>> >>> Hi Oleg and Joe, >>> >>> Kafka 0.8.2.1 >>> >>> Attached is the app log with hostnames scrubbed. >>> >>> Thanks for your help. Much appreciated. >>> >>> -- >>> Josh West <[email protected]<mailto:[email protected]>> >>> Bose Corporation >>> >>> >>> On Sat, 2016-02-20 at 15:46 -0500, Joe Witt wrote: >>> >>> And also what version of Kafka are you using? >>> >>> On Feb 20, 2016 3:37 PM, "Oleg Zhurakousky" >>> <[email protected]<mailto:[email protected]>> >>> wrote: >>> >>> Josh >>> >>> Any chance to attache the app-log or relevant stack trace? >>> >>> Thanks >>> Oleg >>> >>> On Feb 20, 2016, at 3:30 PM, West, Joshua >>> <[email protected]<mailto:[email protected]>> wrote: >>> >>> Hi folks, >>> >>> I've upgraded from Nifi 0.4.1 to 0.5.0 and I am no longer able to use the >>> GetKafka processor. I'm seeing errors like so: >>> >>> 2016-02-20 20:10:14,953 WARN >>> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0] >>> kafka.consumer.ConsumerFetcherThread >>> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0], >>> Error in fetchkafka.consumer.ConsumerFetcherThread$FetchRequest@7b49a642. >>> Possible cause: java.lang.IllegalArgumentException >>> >>> ^ Note the hostname of the server has been scrubbed. >>> >>> My configuration is pretty generic, except that with Zookeeper we use a >>> different root path, so our Zookeeper connect string looks like so: >>> >>> zookeeper-node1:2181,zookeeper-node2:2181,zookeeper-node3:2181/kafka >>> >>> Is anybody else experiencing issues? >>> >>> Thanks. >>> >>> -- >>> Josh West <[email protected]<mailto:[email protected]>> >>> >>> Cloud Architect >>> Bose Corporation >>> >>> >>> >>> >>> <nifi-app.log.kafkaissues.bz2> >>> >>> >>> >>
