RE: Which kafka client to use with spark streaming

2017-12-26 Thread Serkan TAS
Kafka Clients are blocking spark streaming jobs and after a time streaming job queue increases. -Original Message- From: Cody Koeninger [mailto:c...@koeninger.org] Sent: Tuesday, December 26, 2017 6:47 PM To: Diogo Munaro Vieira Cc: Serkan TAS ; user Subject: Re: Which kafka client to

Re: Which kafka client to use with spark streaming

2017-12-26 Thread Cody Koeninger
Do not add a dependency on kafka-clients, the spark-streaming-kafka library has appropriate transitive dependencies. Either version of the spark-streaming-kafka library should work with 1.0 brokers; what problems were you having? On Mon, Dec 25, 2017 at 7:58 PM, Diogo Munaro Vieira wrote: > He

Re: Which kafka client to use with spark streaming

2017-12-25 Thread Diogo Munaro Vieira
Hey Serkan, it depends of your Kafka version... Is it 0.8.2? Em 25 de dez de 2017 06:17, "Serkan TAS" escreveu: > Hi, > > > > Working on spark 2.2.0 cluster and 1.0 kafka brokers. > > > > I was using the library > > "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0" > > > > and had