Java API for statistics of spark job running on yarn

[Serkan TAS]

Hi all,

I am facing and issue for long running spark job on yarn. If there occures some 
bottle neck on hdfs and/or kafka, active batch count increases immidiately.

I am plannning to check the active batch count with java client and create 
alarms for the operations group.

So, is it possible to retrieve active batch count with any java api as we can 
see on monitoring page below ?

/proxy/application_1534314004365_0001/streaming/

Regards,

Serkan






ENERJİSA


serkan@enerjisa.com
www.enerjisa.com.tr

[Description: Description: Açıklama: 
Tick-Tock-Boom-Facebook] [Description: 
Description: Açıklama: Tick-Tock-Boom-GooglePlus] 
  [Description: 
Description: Açıklama: Tick-Tock-Boom-Youtube] 



[cid:image5d27ab.JPG@08344e31.439da30d]






Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler 
içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve 
dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene 
telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu 
iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.

This communication may contain information that is legally privileged, 
confidential or exempt from disclosure. If you are not the intended recipient, 
please note that any dissemination, distribution, or copying of this 
communication is strictly prohibited. Anyone who receives this message in error 
should notify the sender immediately by telephone or by return communication 
and delete it from his or her computer. Only the person who has sent this 
message is responsible for its content.

Invalid Spark URL: spark://HeartbeatReceiver@hostname

[Serkan TAS]

While trying to execute python script with  pycharm on Windows version am 
getting this error.

Anyone has and ideaabout the error ?

Spark version : 2.3.0

py4j.protocol.Py4JJavaError: An error occurred while calling 
None.org.apache.spark.api.java.JavaSparkContext.
: org.apache.spark.SparkException: Invalid Spark URL: spark://HeartbeatReceiver@





ENERJİSA


serkan@enerjisa.com
www.enerjisa.com.tr

[Description: Description: Açıklama: 
Tick-Tock-Boom-Facebook] [Description: 
Description: Açıklama: Tick-Tock-Boom-GooglePlus] 
  [Description: 
Description: Açıklama: Tick-Tock-Boom-Youtube] 



[cid:image4bf178.JPG@f757178a.438d1ea5]






Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler 
içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve 
dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene 
telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu 
iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.

This communication may contain information that is legally privileged, 
confidential or exempt from disclosure. If you are not the intended recipient, 
please note that any dissemination, distribution, or copying of this 
communication is strictly prohibited. Anyone who receives this message in error 
should notify the sender immediately by telephone or by return communication 
and delete it from his or her computer. Only the person who has sent this 
message is responsible for its content.

RE: Which kafka client to use with spark streaming

[Serkan TAS]

Kafka Clients are blocking spark streaming jobs and after a time streaming job 
queue increases.

-Original Message-
From: Cody Koeninger [mailto:c...@koeninger.org]
Sent: Tuesday, December 26, 2017 6:47 PM
To: Diogo Munaro Vieira <diogo.mun...@corp.globo.com>
Cc: Serkan TAS <serkan@enerjisa.com>; user <user@spark.apache.org>
Subject: Re: Which kafka client to use with spark streaming

Do not add a dependency on kafka-clients, the spark-streaming-kafka library has 
appropriate transitive dependencies.

Either version of the spark-streaming-kafka library should work with
1.0 brokers; what problems were you having?



On Mon, Dec 25, 2017 at 7:58 PM, Diogo Munaro Vieira 
<diogo.mun...@corp.globo.com> wrote:
> Hey Serkan, it depends of your Kafka version... Is it 0.8.2?
>
> Em 25 de dez de 2017 06:17, "Serkan TAS" <serkan@enerjisa.com> escreveu:
>>
>> Hi,
>>
>>
>>
>> Working on spark 2.2.0 cluster and 1.0 kafka brokers.
>>
>>
>>
>> I was using the library
>>
>> "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"
>>
>>
>>
>> and had lots of problems during streaming process then downgraded to
>>
>>"org.apache.spark" % "spark-streaming-kafka-0-8_2.11"
>> % "2.2.0"
>>
>>
>>
>> And i know there is also another path which is using kafka-clients
>> jars which has the latest version of 1.0.0
>>
>>
>>
>> 
>>
>> 
>>
>> org.apache.kafka
>>
>> kafka-clients
>>
>> 1.0.0
>>
>> 
>>
>>
>>
>> I am confused which path  is the  right one
>>
>>
>>
>> Thanks…
>>
>>
>>
>>
>>
>>
>> 
>>
>> Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken
>> bilgiler içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu
>> iletiyi çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan
>> kişi, bu durumu derhal gönderene telefonla ya da e-posta ile
>> bildirmeli ve bilgisayarından silmelidir. Bu iletinin içeriğinden
>> yalnızca iletiyi gönderen kişi sorumludur.
>>
>> This communication may contain information that is legally
>> privileged, confidential or exempt from disclosure. If you are not
>> the intended recipient, please note that any dissemination,
>> distribution, or copying of this communication is strictly
>> prohibited. Anyone who receives this message in error should notify
>> the sender immediately by telephone or by return communication and
>> delete it from his or her computer. Only the person who has sent this 
>> message is responsible for its content.



Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler 
içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve 
dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene 
telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu 
iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.

This communication may contain information that is legally privileged, 
confidential or exempt from disclosure. If you are not the intended recipient, 
please note that any dissemination, distribution, or copying of this 
communication is strictly prohibited. Anyone who receives this message in error 
should notify the sender immediately by telephone or by return communication 
and delete it from his or her computer. Only the person who has sent this 
message is responsible for its content.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Which kafka client to use with spark streaming

[Serkan TAS]

Hi,

Working on spark 2.2.0 cluster and 1.0 kafka brokers.

I was using the library
"org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"

and had lots of problems during streaming process then downgraded to
   "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.2.0"

And i know there is also another path which is using kafka-clients jars which 
has the latest version of 1.0.0



org.apache.kafka
kafka-clients
1.0.0


I am confused which path  is the  right one

Thanks…





Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler 
içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve 
dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene 
telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu 
iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.

This communication may contain information that is legally privileged, 
confidential or exempt from disclosure. If you are not the intended recipient, 
please note that any dissemination, distribution, or copying of this 
communication is strictly prohibited. Anyone who receives this message in error 
should notify the sender immediately by telephone or by return communication 
and delete it from his or her computer. Only the person who has sent this 
message is responsible for its content.

RE: Spark application fail wit numRecords error

[Serkan TAS]

Hi,

I checked the following threads but i am still not sure if it is misuse, common 
o a bug.

https://stackoverflow.com/questions/34989539/spark-streaming-from-kafka-has-error-numrecords-must-not-be-negative

https://stackoverflow.com/questions/41319530/why-does-spark-streaming-application-with-kafka-fail-with-requirement-failed-n

https://forums.databricks.com/questions/11055/how-to-resolve-illegalargumentexception-requiremen.html



From: Prem Sure [mailto:sparksure...@gmail.com]
Sent: Wednesday, November 1, 2017 8:11 PM
To: Serkan TAS <serkan@enerjisa.com>
Cc: user@spark.apache.org
Subject: Re: Spark application fail wit numRecords error

Hi, any offset left over for new topic consumption?, case can be the offset is 
beyond current latest offset and cuasing negative.
hoping kafka brokers health is good and are up, this can also be a reason 
sometimes.

On Wed, Nov 1, 2017 at 11:40 AM, Serkan TAS 
<serkan@enerjisa.com<mailto:serkan@enerjisa.com>> wrote:
Hi,

I searched the error in kafka but i think at last, it is related with spark not 
kafka.

Has anyone faced to an exception that is terminating program with error 
"numRecords must not be negative" while streaming  ?

Thanx in advance.

Regards.



Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler 
içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve 
dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene 
telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu 
iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.

This communication may contain information that is legally privileged, 
confidential or exempt from disclosure. If you are not the intended recipient, 
please note that any dissemination, distribution, or copying of this 
communication is strictly prohibited. Anyone who receives this message in error 
should notify the sender immediately by telephone or by return communication 
and delete it from his or her computer. Only the person who has sent this 
message is responsible for its content.




Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler 
içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve 
dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene 
telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu 
iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.

This communication may contain information that is legally privileged, 
confidential or exempt from disclosure. If you are not the intended recipient, 
please note that any dissemination, distribution, or copying of this 
communication is strictly prohibited. Anyone who receives this message in error 
should notify the sender immediately by telephone or by return communication 
and delete it from his or her computer. Only the person who has sent this 
message is responsible for its content.

Spark application fail wit numRecords error

[Serkan TAS]

Hi,

I searched the error in kafka but i think at last, it is related with spark not 
kafka.

Has anyone faced to an exception that is terminating program with error 
"numRecords must not be negative" while streaming  ?

Thanx in advance.

Regards.



Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler 
içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve 
dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene 
telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu 
iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.

This communication may contain information that is legally privileged, 
confidential or exempt from disclosure. If you are not the intended recipient, 
please note that any dissemination, distribution, or copying of this 
communication is strictly prohibited. Anyone who receives this message in error 
should notify the sender immediately by telephone or by return communication 
and delete it from his or her computer. Only the person who has sent this 
message is responsible for its content.