Oleg, We as a community can support folks however we want provided we're doing so within the Apache way and ASF policies.
We can totally help here. Besides we want to make sure this thing is compatible with all systems it should be and this seems totally fair game. Thanks Joe On Tue, Mar 15, 2016 at 1:48 PM, Oleg Zhurakousky <[email protected]> wrote: > Michael > > Unfortunately I am relatively new to ASF culture and so not sure if it would > be appropriate/feasible for ASF distribution of NiFi to support individual > forks of any product/project (ASF gurus feel free to chime in). And with > Kafka it became quite unfortunate since there are many forks out there with > varying levels of compatibility with corresponding ASF version of Kafka. > That said, the best we can do for Kafka is what’s described here > https://issues.apache.org/jira/browse/NIFI-1629 and will go into the > upcoming 0.6 release. Hopefully that will bring some relief. > > Cheers > Oleg > > On Mar 15, 2016, at 1:21 PM, Michael Dyer <[email protected]> > wrote: > > Oleg, > > It's part of the Cloudera distro. Not sure of the lineage beyond that. > Here are a couple of links. > > https://community.cloudera.com/t5/Data-Ingestion-Integration/New-Kafka-0-8-2-0-1-kafka1-3-1-p0-9-Parcel-What-are-the-changes/td-p/30506 > http://archive.cloudera.com/kafka/parcels/1.3.1/ > > Michael > > > On Tue, Mar 15, 2016 at 12:01 PM, Oleg Zhurakousky > <[email protected]> wrote: >> >> Michael >> >> What is KAFKA-0.8.2.0-1.kafka1.3.1.p0.9? I mean where can I get that >> build? >> I guess based on the previous email we’ve tested our code with 3 versions >> of ASF distribution of Kafka and the above version tells me that it may be >> some kind of fork. >> >> Also, we are considering downgrading Kafka dependencies back to the 0.8 >> and of 0.7 provide a new version of Kafka processors that utilize Kafka 0.9 >> new producer/consumer API >> >> Thanks >> Oleg >> >> On Mar 15, 2016, at 11:46 AM, Michael Dyer <[email protected]> >> wrote: >> >> Joe, >> >> I'm seeing a similar issue moving from 0.3.0 to 0.5.1 with >> KAFKA-0.8.2.0-1.kafka1.3.1.p0.9. >> >> I can see the tasks/time counter increment on the processor but no flow >> data ever leaves the processor. There are no errors shown in the bulletin >> board. The app log shows below (repeating). >> >> The rename 0.4.1 nar to 0.5.1 nar trick (restart nifi) works, except that >> the 'batch size' value does not seem to be honored. I have my batch size >> set to 10000, but I'm seeing files written continually (every few seconds) >> with much smaller sizes. I suspect this has to do with the >> `auto.offset.reset` value which defaults to `largest`. From what I have >> read 'smallest' causes the client to start at the beginning which sounds >> like I would be retrieving duplicates. >> >> Renaming 0.3.0 nar to 0.5.1 (restart nifi) restores the original behavior. >> >> yBuffer([[netflow5,0], initOffset 297426 to broker >> BrokerEndPoint(176,n2.foo.bar.com,9092)] ) >> 2016-03-15 07:45:17,237 WARN >> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176] >> kafka.consumer.ConsumerFetcherThread >> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], >> Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@11bd00f6. >> Possible cause: java.lang.IllegalArgumentException >> 2016-03-15 07:45:17,443 INFO >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] >> kafka.utils.VerifiableProperties Verifying properties >> 2016-03-15 07:45:17,443 INFO >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] >> kafka.utils.VerifiableProperties Property client.id is overridden to >> NiFi-b6c67ee3-aa9e-419d-8a57-84ab5e76c017 >> 2016-03-15 07:45:17,443 INFO >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] >> kafka.utils.VerifiableProperties Property metadata.broker.list is overridden >> to n3.foo.bar.com:9092,n2.foo.bar.com:9092,n4.foo.bar.com:9092 >> 2016-03-15 07:45:17,443 INFO >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] >> kafka.utils.VerifiableProperties Property request.timeout.ms is overridden >> to 30000 >> 2016-03-15 07:45:17,443 INFO >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] >> kafka.client.ClientUtils$ Fetching metadata from broker >> BrokerEndPoint(196,n4.foo.bar.com,9092) with correlation id 14 for 1 >> topic(s) Set(netflow5) >> 2016-03-15 07:45:17,443 INFO >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] >> kafka.producer.SyncProducer Connected to n4.foo.bar.com:9092 for producing >> 2016-03-15 07:45:17,444 INFO >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] >> kafka.producer.SyncProducer Disconnecting from n4.foo.bar.com:9092 >> 2016-03-15 07:45:17,444 INFO >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] >> kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1458053114395] >> Added fetcher for partitions ArrayBuffer([[netflow5,0], initOffset 297426 to >> broker BrokerEndPoint(176,n2.foo.bar.com,9092)] ) >> 2016-03-15 07:45:17,449 WARN >> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176] >> kafka.consumer.ConsumerFetcherThread >> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], >> Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@3e69ef73. >> Possible cause: java.lang.IllegalArgumentException >> 2016-03-15 07:45:17,626 INFO [NiFi Web Server-259] >> o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling >> GetKafka[id=4943a24e-af5c-4392-bc45-7008f30674bb] to run >> 2016-03-15 07:45:17,626 INFO [Timer-Driven Process Thread-3] >> k.consumer.ZookeeperConsumerConnector >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0], >> ZKConsumerConnector shutting down >> 2016-03-15 07:45:17,632 INFO [Timer-Driven Process Thread-3] >> kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1458053114395] >> Stopping leader finder thread >> 2016-03-15 07:45:17,633 INFO [Timer-Driven Process Thread-3] >> k.c.ConsumerFetcherManager$LeaderFinderThread >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread], >> Shutting down >> 2016-03-15 07:45:17,634 INFO >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread] >> k.c.ConsumerFetcherManager$LeaderFinderThread >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread], >> Stopped >> 2016-03-15 07:45:17,634 INFO [Timer-Driven Process Thread-3] >> k.c.ConsumerFetcherManager$LeaderFinderThread >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread], >> Shutdown completed >> 2016-03-15 07:45:17,634 INFO [Timer-Driven Process Thread-3] >> kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1458053114395] >> Stopping all fetchers >> 2016-03-15 07:45:17,634 INFO [Timer-Driven Process Thread-3] >> kafka.consumer.ConsumerFetcherThread >> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], >> Shutting down >> 2016-03-15 07:45:17,634 INFO >> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176] >> kafka.consumer.ConsumerFetcherThread >> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], >> Stopped >> 2016-03-15 07:45:17,635 INFO [Timer-Driven Process Thread-3] >> kafka.consumer.ConsumerFetcherThread >> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176], >> Shutdown completed >> 2016-03-15 07:45:17,635 INFO [Timer-Driven Process Thread-3] >> kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1458053114395] >> All connections stopped >> 2016-03-15 07:45:17,635 INFO [ZkClient-EventThread-302-192.168.1.1:2181] >> org.I0Itec.zkclient.ZkEventThread Terminate ZkClient event thread. >> 2016-03-15 07:45:17,638 INFO [Timer-Driven Process Thread-3] >> org.apache.zookeeper.ZooKeeper Session: 0x1535e2aa53b3f61 closed >> 2016-03-15 07:45:17,638 INFO [Timer-Driven Process Thread-3] >> k.consumer.ZookeeperConsumerConnector >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0], >> ZKConsumerConnector shutdown completed in 11 ms >> 2016-03-15 07:45:17,639 INFO [Timer-Driven Process Thread-4-EventThread] >> org.apache.zookeeper.ClientCnxn EventThread shut down >> 2016-03-15 07:45:17,745 INFO [Flow Service Tasks Thread-2] >> o.a.nifi.controller.StandardFlowService Saved flow controller >> org.apache.nifi.controller.FlowController@45d22bd5 // Another save pending = >> false >> 2016-03-15 07:45:18,414 INFO >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0_watcher_executor] >> k.consumer.ZookeeperConsumerConnector >> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0], stopping >> watcher executor thread for consumer >> b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0 >> >> Hope this helps... >> >> Michael >> >> On Mon, Feb 22, 2016 at 4:12 PM, Joe Witt <[email protected]> wrote: >>> >>> Sorry to clarify it working against all three of these at once: >>> - Kafka 0.8.1.1 >>> - Kafka 0.8.2.1 >>> - Kafka 0.9.0.1 >>> >>> Thanks >>> Joe >>> >>> On Mon, Feb 22, 2016 at 4:12 PM, Joe Witt <[email protected]> wrote: >>> > All just as a point of reference we now have a live system running >>> > that is on NiFi 0.5.0 and feeding three versions of Kafka at once: >>> > - 0.8.1 >>> > - 0.8.2.0 >>> > - 0.9.0.1 >>> > >>> > So perhaps there are some particular configurations that cause issues. >>> > Can you share more details about your configuration of Kafka/NiFi and >>> > what sort of security is enabled? >>> > >>> > Thanks >>> > Joe >>> > >>> > On Mon, Feb 22, 2016 at 1:01 PM, Kyle Burke >>> > <[email protected]> wrote: >>> >> I replaced my 0.5.0 kafka nar with the 0.4.1 kakfa nar and it fixed my >>> >> kafka >>> >> issue. I renamed the 0.4.1 nar to be 0.5.0.nar and restart nifi and my >>> >> kafka >>> >> processor started reading my 0.8.2.1 stream. Not elegant but glad it >>> >> worked. >>> >> >>> >> >>> >> Respectfully, >>> >> >>> >> Kyle Burke | Data Science Engineer >>> >> IgnitionOne - Marketing Technology. Simplified. >>> >> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 >>> >> Direct: 404.961.3918 >>> >> >>> >> >>> >> From: Joe Witt >>> >> Reply-To: "[email protected]" >>> >> Date: Sunday, February 21, 2016 at 5:23 PM >>> >> To: "[email protected]" >>> >> Subject: Re: Nifi 0.50 and GetKafka Issues >>> >> >>> >> Yeah the intent is to support 0.8 and 0.9. Will figure something out. >>> >> >>> >> Thanks >>> >> Joe >>> >> >>> >> On Feb 21, 2016 4:47 PM, "West, Joshua" <[email protected]> wrote: >>> >>> >>> >>> Hi Oleg, >>> >>> >>> >>> Hmm -- from what I can tell, this isn't a Zookeeper communication >>> >>> issue. >>> >>> Nifi is able to connect into the Kafka brokers' Zookeeper cluster and >>> >>> retrieve the list of the kafka brokers to connect to. Seems, from >>> >>> the logs, >>> >>> to be a problem when attempting to consume from Kafka itself. >>> >>> >>> >>> I'm guessing that the Kafka 0.9.0 client libraries just are not >>> >>> compatible >>> >>> with Kafka 0.8.2.1 so in order to use Nifi 0.5.0 with Kafka, the >>> >>> Kafka >>> >>> version must be >= 0.9.0. >>> >>> >>> >>> Any change Nifi could add backwards compatible support for Kafka >>> >>> 0.8.2.1 >>> >>> too? Let you choose which client library version, when setting up >>> >>> the >>> >>> GetKafka processor? >>> >>> >>> >>> -- >>> >>> Josh West <[email protected]> >>> >>> Bose Corporation >>> >>> >>> >>> >>> >>> On Sun, 2016-02-21 at 15:02 +0000, Oleg Zhurakousky wrote: >>> >>> >>> >>> Josh >>> >>> >>> >>> Also, keep in mind that there are incompatible property names in >>> >>> Kafka >>> >>> between the 0.7 and 0.8 releases. One of the change that went it was >>> >>> replacing “zk.connectiontimeout.ms” with >>> >>> “zookeeper.connection.timeout.ms”. >>> >>> Not sure if it’s related though, but realizing that 0.4.1 was relying >>> >>> on >>> >>> this property it’s value was completely ignored with 0.8 client >>> >>> libraries >>> >>> (you could actually see the WARN message to that effect) and now it >>> >>> is not >>> >>> ignored, so take a look and see if tinkering with its value changes >>> >>> something. >>> >>> >>> >>> Cheers >>> >>> Oleg >>> >>> >>> >>> On Feb 20, 2016, at 6:47 PM, Oleg Zhurakousky >>> >>> <[email protected]> wrote: >>> >>> >>> >>> Josh >>> >>> >>> >>> The only change that ’s went and relevant to your issue is the fact >>> >>> that >>> >>> we’ve upgraded client libraries to Kafka 0.9 and between 0.8 and 0.9 >>> >>> Kafka >>> >>> introduced wire protocol changes that break compatibility. >>> >>> I am still digging so stay tuned. >>> >>> >>> >>> Oleg >>> >>> >>> >>> On Feb 20, 2016, at 4:10 PM, West, Joshua <[email protected]> wrote: >>> >>> >>> >>> Hi Oleg and Joe, >>> >>> >>> >>> Kafka 0.8.2.1 >>> >>> >>> >>> Attached is the app log with hostnames scrubbed. >>> >>> >>> >>> Thanks for your help. Much appreciated. >>> >>> >>> >>> -- >>> >>> Josh West <[email protected]> >>> >>> Bose Corporation >>> >>> >>> >>> >>> >>> On Sat, 2016-02-20 at 15:46 -0500, Joe Witt wrote: >>> >>> >>> >>> And also what version of Kafka are you using? >>> >>> >>> >>> On Feb 20, 2016 3:37 PM, "Oleg Zhurakousky" >>> >>> <[email protected]> >>> >>> wrote: >>> >>> >>> >>> Josh >>> >>> >>> >>> Any chance to attache the app-log or relevant stack trace? >>> >>> >>> >>> Thanks >>> >>> Oleg >>> >>> >>> >>> On Feb 20, 2016, at 3:30 PM, West, Joshua <[email protected]> wrote: >>> >>> >>> >>> Hi folks, >>> >>> >>> >>> I've upgraded from Nifi 0.4.1 to 0.5.0 and I am no longer able to use >>> >>> the >>> >>> GetKafka processor. I'm seeing errors like so: >>> >>> >>> >>> 2016-02-20 20:10:14,953 WARN >>> >>> >>> >>> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0] >>> >>> kafka.consumer.ConsumerFetcherThread >>> >>> >>> >>> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0], >>> >>> Error in >>> >>> fetchkafka.consumer.ConsumerFetcherThread$FetchRequest@7b49a642. >>> >>> Possible cause: java.lang.IllegalArgumentException >>> >>> >>> >>> ^ Note the hostname of the server has been scrubbed. >>> >>> >>> >>> My configuration is pretty generic, except that with Zookeeper we use >>> >>> a >>> >>> different root path, so our Zookeeper connect string looks like so: >>> >>> >>> >>> zookeeper-node1:2181,zookeeper-node2:2181,zookeeper-node3:2181/kafka >>> >>> >>> >>> Is anybody else experiencing issues? >>> >>> >>> >>> Thanks. >>> >>> >>> >>> -- >>> >>> Josh West <[email protected]> >>> >>> >>> >>> Cloud Architect >>> >>> Bose Corporation >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> <nifi-app.log.kafkaissues.bz2> >>> >>> >>> >>> >>> >>> >>> >> >> >> >> > >
