Thanks Emanuele Does this all apply to Flink as well? Cheers
Sent from my iPhone > On Aug 11, 2016, at 10:48 PM, Emanuele Cesena <[email protected]> wrote: > > Hi Amir, > > Which version of Spark are you using, 1.6 or 2? > > On hadoop I don’t think there’s any issue and I’d go with the latest 2.7 (or > 2.6 if you have any special requirements). > > To the best of my knowledge: > - Spark 2 works with Kafka 0.9+ > - Spark 1.6 requires Kafka 0.8.x — although some distro can have different > versions, for instance we’re using CDH 5.7 that ships Spark 1.6 working with > Kafka 0.9. > > If you use/plan to use Beam’s KafkaIO then it requires Kafka 0.9+ (I’d > personally go with 0.10), but the last time I tried it with Flink it wasn’t > fully working because of this: > https://issues.apache.org/jira/browse/BEAM-102 > (Aljoscha told me he was working on it and we can expect a patch soon) > > This is all I know. > > Best, > > >> On Aug 11, 2016, at 5:16 PM, amir bahmanyari <[email protected]> wrote: >> >> Hi Colleagues, >> I am rebuilding my Beam lab servers with the latest clusters Flink & >> eventually Spark. >> Whats the appropriate versions of Kafka & Hadoop to install so there wont be >> any potential short ckt? >> >> Thanks >> Amir- > > -- > Emanuele Cesena, Data Eng. > http://www.shopkick.com > > Il corpo non ha ideali > > > >
