Hi Amir, Which version of Spark are you using, 1.6 or 2?
On hadoop I don’t think there’s any issue and I’d go with the latest 2.7 (or 2.6 if you have any special requirements). To the best of my knowledge: - Spark 2 works with Kafka 0.9+ - Spark 1.6 requires Kafka 0.8.x — although some distro can have different versions, for instance we’re using CDH 5.7 that ships Spark 1.6 working with Kafka 0.9. If you use/plan to use Beam’s KafkaIO then it requires Kafka 0.9+ (I’d personally go with 0.10), but the last time I tried it with Flink it wasn’t fully working because of this: https://issues.apache.org/jira/browse/BEAM-102 (Aljoscha told me he was working on it and we can expect a patch soon) This is all I know. Best, > On Aug 11, 2016, at 5:16 PM, amir bahmanyari <[email protected]> wrote: > > Hi Colleagues, > I am rebuilding my Beam lab servers with the latest clusters Flink & > eventually Spark. > Whats the appropriate versions of Kafka & Hadoop to install so there wont be > any potential short ckt? > > Thanks > Amir- -- Emanuele Cesena, Data Eng. http://www.shopkick.com Il corpo non ha ideali
