Hi, The last part (the jira issue) is about Flink. (Flink itself should have support for all Kafka versions, but I don’t think that’s the recommended way to go with Beam.)
Best, > On Aug 11, 2016, at 11:19 PM, Amir Bahmanyari <[email protected]> wrote: > > Thanks Emanuele > Does this all apply to Flink as well? > Cheers > > > Sent from my iPhone > >> On Aug 11, 2016, at 10:48 PM, Emanuele Cesena <[email protected]> wrote: >> >> Hi Amir, >> >> Which version of Spark are you using, 1.6 or 2? >> >> On hadoop I don’t think there’s any issue and I’d go with the latest 2.7 (or >> 2.6 if you have any special requirements). >> >> To the best of my knowledge: >> - Spark 2 works with Kafka 0.9+ >> - Spark 1.6 requires Kafka 0.8.x — although some distro can have different >> versions, for instance we’re using CDH 5.7 that ships Spark 1.6 working with >> Kafka 0.9. >> >> If you use/plan to use Beam’s KafkaIO then it requires Kafka 0.9+ (I’d >> personally go with 0.10), but the last time I tried it with Flink it wasn’t >> fully working because of this: >> https://issues.apache.org/jira/browse/BEAM-102 >> (Aljoscha told me he was working on it and we can expect a patch soon) >> >> This is all I know. >> >> Best, >> >> >>> On Aug 11, 2016, at 5:16 PM, amir bahmanyari <[email protected]> wrote: >>> >>> Hi Colleagues, >>> I am rebuilding my Beam lab servers with the latest clusters Flink & >>> eventually Spark. >>> Whats the appropriate versions of Kafka & Hadoop to install so there wont >>> be any potential short ckt? >>> >>> Thanks >>> Amir- >> >> -- >> Emanuele Cesena, Data Eng. >> http://www.shopkick.com >> >> Il corpo non ha ideali >> >> >> >> > -- Emanuele Cesena, Data Eng. http://www.shopkick.com Il corpo non ha ideali
