RE: Which kafka client to use with spark streaming

[Serkan TAS]

Kafka Clients are blocking spark streaming jobs and after a time streaming job 
queue increases.

-Original Message-
From: Cody Koeninger [mailto:c...@koeninger.org]
Sent: Tuesday, December 26, 2017 6:47 PM
To: Diogo Munaro Vieira <diogo.mun...@corp.globo.com>
Cc: Serkan TAS <serkan@enerjisa.com>; user <user@spark.apache.org>
Subject: Re: Which kafka client to use with spark streaming

Do not add a dependency on kafka-clients, the spark-streaming-kafka library has 
appropriate transitive dependencies.

Either version of the spark-streaming-kafka library should work with
1.0 brokers; what problems were you having?



On Mon, Dec 25, 2017 at 7:58 PM, Diogo Munaro Vieira 
<diogo.mun...@corp.globo.com> wrote:
> Hey Serkan, it depends of your Kafka version... Is it 0.8.2?
>
> Em 25 de dez de 2017 06:17, "Serkan TAS" <serkan@enerjisa.com> escreveu:
>>
>> Hi,
>>
>>
>>
>> Working on spark 2.2.0 cluster and 1.0 kafka brokers.
>>
>>
>>
>> I was using the library
>>
>> "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"
>>
>>
>>
>> and had lots of problems during streaming process then downgraded to
>>
>>"org.apache.spark" % "spark-streaming-kafka-0-8_2.11"
>> % "2.2.0"
>>
>>
>>
>> And i know there is also another path which is using kafka-clients
>> jars which has the latest version of 1.0.0
>>
>>
>>
>> 
>>
>> 
>>
>> org.apache.kafka
>>
>> kafka-clients
>>
>> 1.0.0
>>
>> 
>>
>>
>>
>> I am confused which path  is the  right one
>>
>>
>>
>> Thanks…
>>
>>
>>
>>
>>
>>
>> 
>>
>> Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken
>> bilgiler içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu
>> iletiyi çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan
>> kişi, bu durumu derhal gönderene telefonla ya da e-posta ile
>> bildirmeli ve bilgisayarından silmelidir. Bu iletinin içeriğinden
>> yalnızca iletiyi gönderen kişi sorumludur.
>>
>> This communication may contain information that is legally
>> privileged, confidential or exempt from disclosure. If you are not
>> the intended recipient, please note that any dissemination,
>> distribution, or copying of this communication is strictly
>> prohibited. Anyone who receives this message in error should notify
>> the sender immediately by telephone or by return communication and
>> delete it from his or her computer. Only the person who has sent this 
>> message is responsible for its content.



Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler 
içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve 
dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene 
telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu 
iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.

This communication may contain information that is legally privileged, 
confidential or exempt from disclosure. If you are not the intended recipient, 
please note that any dissemination, distribution, or copying of this 
communication is strictly prohibited. Anyone who receives this message in error 
should notify the sender immediately by telephone or by return communication 
and delete it from his or her computer. Only the person who has sent this 
message is responsible for its content.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Which kafka client to use with spark streaming

[Cody Koeninger]

Do not add a dependency on kafka-clients, the spark-streaming-kafka
library has appropriate transitive dependencies.

Either version of the spark-streaming-kafka library should work with
1.0 brokers; what problems were you having?



On Mon, Dec 25, 2017 at 7:58 PM, Diogo Munaro Vieira
 wrote:
> Hey Serkan, it depends of your Kafka version... Is it 0.8.2?
>
> Em 25 de dez de 2017 06:17, "Serkan TAS"  escreveu:
>>
>> Hi,
>>
>>
>>
>> Working on spark 2.2.0 cluster and 1.0 kafka brokers.
>>
>>
>>
>> I was using the library
>>
>> "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"
>>
>>
>>
>> and had lots of problems during streaming process then downgraded to
>>
>>"org.apache.spark" % "spark-streaming-kafka-0-8_2.11" %
>> "2.2.0"
>>
>>
>>
>> And i know there is also another path which is using kafka-clients jars
>> which has the latest version of 1.0.0
>>
>>
>>
>> 
>>
>> 
>>
>> org.apache.kafka
>>
>> kafka-clients
>>
>> 1.0.0
>>
>> 
>>
>>
>>
>> I am confused which path  is the  right one
>>
>>
>>
>> Thanks…
>>
>>
>>
>>
>>
>>
>> 
>>
>> Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler
>> içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak
>> ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal
>> gönderene telefonla ya da e-posta ile bildirmeli ve bilgisayarından
>> silmelidir. Bu iletinin içeriğinden yalnızca iletiyi gönderen kişi
>> sorumludur.
>>
>> This communication may contain information that is legally privileged,
>> confidential or exempt from disclosure. If you are not the intended
>> recipient, please note that any dissemination, distribution, or copying of
>> this communication is strictly prohibited. Anyone who receives this message
>> in error should notify the sender immediately by telephone or by return
>> communication and delete it from his or her computer. Only the person who
>> has sent this message is responsible for its content.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Which kafka client to use with spark streaming

[Diogo Munaro Vieira]

Hey Serkan, it depends of your Kafka version... Is it 0.8.2?

Em 25 de dez de 2017 06:17, "Serkan TAS"  escreveu:

> Hi,
>
>
>
> Working on spark 2.2.0 cluster and 1.0 kafka brokers.
>
>
>
> I was using the library
>
> "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"
>
>
>
> and had lots of problems during streaming process then downgraded to
>
>"org.apache.spark" % "spark-streaming-kafka-0-8_2.11" %
> "2.2.0"
>
>
>
> And i know there is also another path which is using kafka-clients jars
> which has the latest version of 1.0.0
>
>
>
> 
>
> 
>
> org.apache.kafka
>
> kafka-clients
>
> 1.0.0
>
> 
>
>
>
> I am confused which path  is the  right one
>
>
>
> Thanks…
>
>
>
>
>
> --
>
> Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler
> içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi
> çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu
> derhal gönderene telefonla ya da e-posta ile bildirmeli ve bilgisayarından
> silmelidir. Bu iletinin içeriğinden yalnızca iletiyi gönderen kişi
> sorumludur.
>
> This communication may contain information that is legally privileged,
> confidential or exempt from disclosure. If you are not the intended
> recipient, please note that any dissemination, distribution, or copying of
> this communication is strictly prohibited. Anyone who receives this message
> in error should notify the sender immediately by telephone or by return
> communication and delete it from his or her computer. Only the person who
> has sent this message is responsible for its content.
>

Which kafka client to use with spark streaming

[Serkan TAS]

Hi,

Working on spark 2.2.0 cluster and 1.0 kafka brokers.

I was using the library
"org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"

and had lots of problems during streaming process then downgraded to
   "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.2.0"

And i know there is also another path which is using kafka-clients jars which 
has the latest version of 1.0.0



org.apache.kafka
kafka-clients
1.0.0


I am confused which path  is the  right one

Thanks…





Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler 
içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve 
dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene 
telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu 
iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.

This communication may contain information that is legally privileged, 
confidential or exempt from disclosure. If you are not the intended recipient, 
please note that any dissemination, distribution, or copying of this 
communication is strictly prohibited. Anyone who receives this message in error 
should notify the sender immediately by telephone or by return communication 
and delete it from his or her computer. Only the person who has sent this 
message is responsible for its content.