Hi Fawze,
Thank you for the link. But that is exactly what I am doing.
I think this is related to
yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage
setting.
When the disk utilization exceeds this setting, the node is marked
unhealthy.
Other than increasing the default
http://shzhangji.com/blog/2015/05/31/spark-streaming-logging-configuration/
On Wed, Dec 26, 2018 at 1:05 AM shyla deshpande
wrote:
> Please point me to any documentation if available. Thanks
>
> On Tue, Dec 18, 2018 at 11:10 AM shyla deshpande
> wrote:
>
>> Is there a way to do this without
Please point me to any documentation if available. Thanks
On Tue, Dec 18, 2018 at 11:10 AM shyla deshpande
wrote:
> Is there a way to do this without stopping the streaming application in
> yarn cluster mode?
>
> On Mon, Dec 17, 2018 at 4:42 PM shyla deshpande
> wrote:
>
>> I get the ERROR
>>
Is there a way to do this without stopping the streaming application in
yarn cluster mode?
On Mon, Dec 17, 2018 at 4:42 PM shyla deshpande
wrote:
> I get the ERROR
> 1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad:
> /var/log/hadoop-yarn/containers
>
> Is there a way to clean up these
I get the ERROR
1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad:
/var/log/hadoop-yarn/containers
Is there a way to clean up these directories while the spark streaming
application is running?
Thanks
Two weeks ago I have published a blogpost about our experiences running 24/7
Spark Streaming applications on YARN in production:
https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/
<https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/>
Amongst
I am running spark streaming with Yarn -
*spark-submit --master yarn --deploy-mode cluster --num-executors 2
> --executor-memory 8g --driver-memory 2g --executor-cores 8 ..*
>
I am consuming Kafka through DireactStream approach (No receiver). I have 2
topics (each with 3 partition
Any update on this guys ?
On Wed, Dec 28, 2016 at 10:19 AM, Nishant Kumar <nishant.ku...@applift.com>
wrote:
> I have updated my question:
>
> http://stackoverflow.com/questions/41345552/spark-
> streaming-with-yarn-executors-not-fully-utilized
>
> On Wed, Dec 28, 2016 a
I have updated my question:
http://stackoverflow.com/questions/41345552/spark-streaming-with-yarn-executors-not-fully-utilized
On Wed, Dec 28, 2016 at 9:49 AM, Nishant Kumar <nishant.ku...@applift.com>
wrote:
> Hi,
>
> I am running spark streaming with Yarn with -
>
> *
Hi,
I am running spark streaming with Yarn with -
*spark-submit --master yarn --deploy-mode cluster --num-executors 2
--executor-memory 8g --driver-memory 2g --executor-cores 8 ..*
I am consuming Kafka through DireactStream approach (No receiver). I have 2
topics (each with 3 partitions).
I
the deployment mode to yarn-client.
>
> Thanks
> Deepak
>
>
> On Fri, May 13, 2016 at 10:17 AM, Rakesh H (Marketing Platform-BLR) <
> rakes...@flipkart.com> wrote:
>
>> Ping!!
>> Has anybody tested graceful shutdown of a spark streaming in yarn-cluster
>> m
Platform-BLR) <
rakes...@flipkart.com> wrote:
> Ping!!
> Has anybody tested graceful shutdown of a spark streaming in yarn-cluster
> mode?It looks like a defect to me.
>
>
> On Thu, May 12, 2016 at 12:53 PM Rakesh H (Marketing Platform-BLR) <
> rakes...@flipkart.com>
Ping!!
Has anybody tested graceful shutdown of a spark streaming in yarn-cluster
mode?It looks like a defect to me.
On Thu, May 12, 2016 at 12:53 PM Rakesh H (Marketing Platform-BLR) <
rakes...@flipkart.com> wrote:
> We are on spark 1.5.1
> Above change was to add a shutdown
>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, May 12, 2016 at 11:45 AM Deepak Sharma <deepakmc...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Rakesh
>>>>> Did you trie
pakmc...@gmail.com>
>>> wrote:
>>>
>>>> Hi Rakesh
>>>> Did you tried setting *spark.streaming.stopGracefullyOnShutdown to
>>>> true *for your spark configuration instance?
>>>> If not try this , and let us know if this helps.
>>>
2016 at 11:45 AM Deepak Sharma <deepakmc...@gmail.com>
>> wrote:
>>
>>> Hi Rakesh
>>> Did you tried setting *spark.streaming.stopGracefullyOnShutdown to true
>>> *for your spark configuration instance?
>>> If not try this , and let us know if thi
tting *spark.streaming.stopGracefullyOnShutdown to true *for
>> your spark configuration instance?
>> If not try this , and let us know if this helps.
>>
>> Thanks
>> Deepak
>>
>> On Thu, May 12, 2016 at 11:42 AM, Rakesh H (Marketing Platform-BLR) <
>>
try this , and let us know if this helps.
>
> Thanks
> Deepak
>
> On Thu, May 12, 2016 at 11:42 AM, Rakesh H (Marketing Platform-BLR) <
> rakes...@flipkart.com> wrote:
>
>> Issue i am having is similar to the one mentioned here :
>>
>> http://stackoverflow.com
; Issue i am having is similar to the one mentioned here :
>
> http://stackoverflow.com/questions/36911442/how-to-stop-gracefully-a-spark-streaming-application-on-yarn
>
> I am creating a rdd from sequence of 1 to 300 and creating streaming RDD
> out of it.
>
> val rdd = ssc.spa
Issue i am having is similar to the one mentioned here :
http://stackoverflow.com/questions/36911442/how-to-stop-gracefully-a-spark-streaming-application-on-yarn
I am creating a rdd from sequence of 1 to 300 and creating streaming RDD
out of it.
val rdd = ssc.sparkContext.parallelize(1 to 300
4 partitions.
- Mail original -
De: "Dibyendu Bhattacharya" <dibyendu.bhattach...@gmail.com>
À: "Nicolas Biau" <nib...@free.fr>
Cc: "Cody Koeninger" <c...@koeninger.org>, "user" <user@spark.apache.org>
Envoyé: Dimanche 4
endu.bhattach...@gmail.com>
À: nib...@free.fr
Cc: "Cody Koeninger" <c...@koeninger.org>, "user" <user@spark.apache.org>
Envoyé: Vendredi 2 Octobre 2015 18:21:35
Objet: Re: Spark Streaming over YARN
If your Kafka topic has 4 partitions , and if you specify 4 Receiv
g>, "user" <user@spark.apache.org>
> Envoyé: Vendredi 2 Octobre 2015 18:21:35
> Objet: Re: Spark Streaming over YARN
>
>
> If your Kafka topic has 4 partitions , and if you specify 4 Receivers,
> messages from each partitions are received by a dedicated recei
uot;user" <user@spark.apache.org>
> Envoyé: Vendredi 2 Octobre 2015 17:43:41
> Objet: Re: Spark Streaming over YARN
>
>
> If you're using the receiver based implementation, and want more
> parallelism, you have to create multiple streams and union them together.
>
>
r a spark job failure)
>
>
> - Mail original -
> De: "Cody Koeninger" <c...@koeninger.org>
> À: "Nicolas Biau" <nib...@free.fr>
> Cc: "user" <user@spark.apache.org>
> Envoyé: Vendredi 2 Octobre 2015 17:43:41
> Objet: Re: Spa
Hello,
I have a job receiving data from kafka (4 partitions) and persisting data
inside MongoDB.
It works fine, but when I deploy it inside YARN cluster (4 nodes with 2 cores)
only on node is receiving all the kafka partitions and only one node is
processing my RDD treatment (foreach function)
If you're using the receiver based implementation, and want more
parallelism, you have to create multiple streams and union them together.
Or use the direct stream.
On Fri, Oct 2, 2015 at 10:40 AM, wrote:
> Hello,
> I have a job receiving data from kafka (4 partitions) and
iginal -
De: "Cody Koeninger" <c...@koeninger.org>
À: "Nicolas Biau" <nib...@free.fr>
Cc: "user" <user@spark.apache.org>
Envoyé: Vendredi 2 Octobre 2015 17:43:41
Objet: Re: Spark Streaming over YARN
If you're using the receiver based implemen
ree.fr
Cc: "Cody Koeninger" <c...@koeninger.org>, "user" <user@spark.apache.org>
Envoyé: Vendredi 2 Octobre 2015 18:01:59
Objet: Re: Spark Streaming over YARN
Hi,
If you need to use Receiver based approach , you can try this one :
https://github.com/dibbhatt/ka
e.fr
> Cc: "Cody Koeninger" <c...@koeninger.org>, "user" <user@spark.apache.org>
> Envoyé: Vendredi 2 Octobre 2015 18:01:59
> Objet: Re: Spark Streaming over YARN
>
>
> Hi,
>
>
> If you need to use Receiver based approach , you can try this o
De: "Cody Koeninger" <c...@koeninger.org>
> À: "Nicolas Biau" <nib...@free.fr>
> Cc: "user" <user@spark.apache.org>
> Envoyé: Vendredi 2 Octobre 2015 18:29:09
> Objet: Re: Spark Streaming over YARN
>
>
> Neither of those statements are tr
...@free.fr>
Cc: "user" <user@spark.apache.org>
Envoyé: Vendredi 2 Octobre 2015 18:29:09
Objet: Re: Spark Streaming over YARN
Neither of those statements are true.
You need more receivers if you want more parallelism.
You don't have to manage offset positioning with the direct
:00 Kyle Lin kylelin2...@gmail.com:
Hi all
My environment info
Hadoop release version: HDP 2.1
Kakfa: 0.8.1.2.1.4.0
Spark: 1.1.0
My question:
I ran Spark streaming program on YARN. My Spark streaming program will
read data from Kafka and doing some processing. But, I found
/sets
From: Kyle Lin kylelin2...@gmail.com
To: user@spark.apache.org user@spark.apache.org
Date: 2015/04/30 14:39
Subject:The Processing loading of Spark streaming on YARN is not in
balance
Hi all
My environment info
Hadoop release version: HDP 2.1
Kakfa
Hi all
My environment info
Hadoop release version: HDP 2.1
Kakfa: 0.8.1.2.1.4.0
Spark: 1.1.0
My question:
I ran Spark streaming program on YARN. My Spark streaming program will
read data from Kafka and doing some processing. But, I found there is
always only ONE executor under processing
---2015/04/30 14:39:32---Hi all My
environment info
From: Kyle Lin kylelin2...@gmail.com
To: user@spark.apache.org user@spark.apache.org
Date: 2015/04/30 14:39
Subject: The Processing loading of Spark streaming on YARN is not in
balance
--
Hi all
My
My environment info
Hadoop release version: HDP 2.1
Kakfa: 0.8.1.2.1.4.0
Spark: 1.1.0
My question:
I ran Spark streaming program on YARN. My Spark streaming program
will read data from Kafka and doing some processing. But, I found there is
always only ONE executor under processing
I’m looking at various HA scenarios with Spark streaming. We’re currently
running a Spark streaming job that is intended to be long-lived, 24/7. We see
that if we kill node managers that are hosting Spark workers, new node managers
assume execution of the jobs that were running on the
have you fixed this issue ?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-on-Yarn-Input-from-Flume-tp11755p22055.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-on-Yarn-tp19093.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr
normally .But if I
run this app on yarn,no matter on yarn-client or yarn-cluster. This app can
not receive data from flume,and I check the net stat,I find the port which
connect to flume,didn't be listened.
I am not sure,if spark streaming run on Yarn,will affect connect to flume.
And then I
which
connect to flume,didn't be listened.
I am not sure,if spark streaming run on Yarn,will affect connect to flume.
And then I run the FlumeEventCount example, when I run this example on
Yarn, it also can not receive data from flume.
And I will be very pleasure if some one can help me
Hi guys,
Not sure if you have similar issues. Did not find relevant tickets in
JIRA. When I deploy the Spark Streaming to YARN, I have following two
issues:
1. The UI port is random. It is not default 4040. I have to look at the
container's log to check the UI port. Is this suppose
not find relevant tickets in
JIRA. When I deploy the Spark Streaming to YARN, I have following two
issues:
1. The UI port is random. It is not default 4040. I have to look at the
container's log to check the UI port. Is this suppose to be this way?
2. Most of the time, the UI does not work
2014-07-07 11:20 GMT-07:00 Yan Fang yanfang...@gmail.com:
Hi guys,
Not sure if you have similar issues. Did not find relevant tickets in
JIRA. When I deploy the Spark Streaming to YARN, I have following two
issues:
1. The UI port is random. It is not default 4040. I have to look
yanfang...@gmail.com:
Hi guys,
Not sure if you have similar issues. Did not find relevant tickets in
JIRA. When I deploy the Spark Streaming to YARN, I have following two
issues:
1. The UI port is random. It is not default 4040. I have to look at the
container's log to check the UI port
, not the executor containers. This is because SparkUI belongs to
the SparkContext, which only exists on the driver.
Andrew
2014-07-07 11:20 GMT-07:00 Yan Fang yanfang...@gmail.com:
Hi guys,
Not sure if you have similar issues. Did not find relevant tickets in
JIRA. When I deploy the Spark Streaming
GMT-07:00 Yan Fang yanfang...@gmail.com:
Hi guys,
Not sure if you have similar issues. Did not find relevant tickets in
JIRA. When I deploy the Spark Streaming to YARN, I have following two
issues:
1. The UI port is random. It is not default 4040. I have to look at the
container's log
48 matches
Mail list logo