Re: How to clean up logs-dirs and local-dirs of running spark streaming in yarn cluster mode

Hi Fawze, Thank you for the link. But that is exactly what I am doing. I think this is related to yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage setting. When the disk utilization exceeds this setting, the node is marked unhealthy. Other than increasing the default

Re: How to clean up logs-dirs and local-dirs of running spark streaming in yarn cluster mode

http://shzhangji.com/blog/2015/05/31/spark-streaming-logging-configuration/ On Wed, Dec 26, 2018 at 1:05 AM shyla deshpande wrote: > Please point me to any documentation if available. Thanks > > On Tue, Dec 18, 2018 at 11:10 AM shyla deshpande > wrote: > >> Is there a way to do this without

Re: How to clean up logs-dirs and local-dirs of running spark streaming in yarn cluster mode

Please point me to any documentation if available. Thanks On Tue, Dec 18, 2018 at 11:10 AM shyla deshpande wrote: > Is there a way to do this without stopping the streaming application in > yarn cluster mode? > > On Mon, Dec 17, 2018 at 4:42 PM shyla deshpande > wrote: > >> I get the ERROR >>

Re: How to clean up logs-dirs and local-dirs of running spark streaming in yarn cluster mode

Is there a way to do this without stopping the streaming application in yarn cluster mode? On Mon, Dec 17, 2018 at 4:42 PM shyla deshpande wrote: > I get the ERROR > 1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: > /var/log/hadoop-yarn/containers > > Is there a way to clean up these

How to clean up logs-dirs and local-dirs of running spark streaming in yarn cluster mode

I get the ERROR 1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers Is there a way to clean up these directories while the spark streaming application is running? Thanks

24/7 Spark Streaming on YARN in Production

Two weeks ago I have published a blogpost about our experiences running 24/7 Spark Streaming applications on YARN in production: https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/ <https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/> Amongst

[Spark streaming 1.6.0] Spark streaming with Yarn: executors not fully utilized

I am running spark streaming with Yarn - *spark-submit --master yarn --deploy-mode cluster --num-executors 2 > --executor-memory 8g --driver-memory 2g --executor-cores 8 ..* > I am consuming Kafka through DireactStream approach (No receiver). I have 2 topics (each with 3 partition

Re: Spark streaming with Yarn: executors not fully utilized

Any update on this guys ? On Wed, Dec 28, 2016 at 10:19 AM, Nishant Kumar <nishant.ku...@applift.com> wrote: > I have updated my question: > > http://stackoverflow.com/questions/41345552/spark- > streaming-with-yarn-executors-not-fully-utilized > > On Wed, Dec 28, 2016 a

Re: Spark streaming with Yarn: executors not fully utilized

I have updated my question: http://stackoverflow.com/questions/41345552/spark-streaming-with-yarn-executors-not-fully-utilized On Wed, Dec 28, 2016 at 9:49 AM, Nishant Kumar <nishant.ku...@applift.com> wrote: > Hi, > > I am running spark streaming with Yarn with - > > *

Spark streaming with Yarn: executors not fully utilized

Hi, I am running spark streaming with Yarn with - *spark-submit --master yarn --deploy-mode cluster --num-executors 2 --executor-memory 8g --driver-memory 2g --executor-cores 8 ..* I am consuming Kafka through DireactStream approach (No receiver). I have 2 topics (each with 3 partitions). I

Re: Graceful shutdown of spark streaming on yarn

the deployment mode to yarn-client. > > Thanks > Deepak > > > On Fri, May 13, 2016 at 10:17 AM, Rakesh H (Marketing Platform-BLR) < > rakes...@flipkart.com> wrote: > >> Ping!! >> Has anybody tested graceful shutdown of a spark streaming in yarn-cluster >> m

Re: Graceful shutdown of spark streaming on yarn

Platform-BLR) < rakes...@flipkart.com> wrote: > Ping!! > Has anybody tested graceful shutdown of a spark streaming in yarn-cluster > mode?It looks like a defect to me. > > > On Thu, May 12, 2016 at 12:53 PM Rakesh H (Marketing Platform-BLR) < > rakes...@flipkart.com>

Re: Graceful shutdown of spark streaming on yarn

Ping!! Has anybody tested graceful shutdown of a spark streaming in yarn-cluster mode?It looks like a defect to me. On Thu, May 12, 2016 at 12:53 PM Rakesh H (Marketing Platform-BLR) < rakes...@flipkart.com> wrote: > We are on spark 1.5.1 > Above change was to add a shutdown

Re: Graceful shutdown of spark streaming on yarn

>> >>>> >>>> >>>> >>>> >>>> >>>> On Thu, May 12, 2016 at 11:45 AM Deepak Sharma <deepakmc...@gmail.com> >>>> wrote: >>>> >>>>> Hi Rakesh >>>>> Did you trie

Re: Graceful shutdown of spark streaming on yarn

pakmc...@gmail.com> >>> wrote: >>> >>>> Hi Rakesh >>>> Did you tried setting *spark.streaming.stopGracefullyOnShutdown to >>>> true *for your spark configuration instance? >>>> If not try this , and let us know if this helps. >>>

Re: Graceful shutdown of spark streaming on yarn

2016 at 11:45 AM Deepak Sharma <deepakmc...@gmail.com> >> wrote: >> >>> Hi Rakesh >>> Did you tried setting *spark.streaming.stopGracefullyOnShutdown to true >>> *for your spark configuration instance? >>> If not try this , and let us know if thi

Re: Graceful shutdown of spark streaming on yarn

tting *spark.streaming.stopGracefullyOnShutdown to true *for >> your spark configuration instance? >> If not try this , and let us know if this helps. >> >> Thanks >> Deepak >> >> On Thu, May 12, 2016 at 11:42 AM, Rakesh H (Marketing Platform-BLR) < >>

Re: Graceful shutdown of spark streaming on yarn

try this , and let us know if this helps. > > Thanks > Deepak > > On Thu, May 12, 2016 at 11:42 AM, Rakesh H (Marketing Platform-BLR) < > rakes...@flipkart.com> wrote: > >> Issue i am having is similar to the one mentioned here : >> >> http://stackoverflow.com

Re: Graceful shutdown of spark streaming on yarn

; Issue i am having is similar to the one mentioned here : > > http://stackoverflow.com/questions/36911442/how-to-stop-gracefully-a-spark-streaming-application-on-yarn > > I am creating a rdd from sequence of 1 to 300 and creating streaming RDD > out of it. > > val rdd = ssc.spa

Graceful shutdown of spark streaming on yarn

Issue i am having is similar to the one mentioned here : http://stackoverflow.com/questions/36911442/how-to-stop-gracefully-a-spark-streaming-application-on-yarn I am creating a rdd from sequence of 1 to 300 and creating streaming RDD out of it. val rdd = ssc.sparkContext.parallelize(1 to 300

Re: Spark Streaming over YARN

4 partitions. - Mail original - De: "Dibyendu Bhattacharya" <dibyendu.bhattach...@gmail.com> À: "Nicolas Biau" <nib...@free.fr> Cc: "Cody Koeninger" <c...@koeninger.org>, "user" <user@spark.apache.org> Envoyé: Dimanche 4

Re: Spark Streaming over YARN

endu.bhattach...@gmail.com> À: nib...@free.fr Cc: "Cody Koeninger" <c...@koeninger.org>, "user" <user@spark.apache.org> Envoyé: Vendredi 2 Octobre 2015 18:21:35 Objet: Re: Spark Streaming over YARN If your Kafka topic has 4 partitions , and if you specify 4 Receiv

Re: Spark Streaming over YARN

g>, "user" <user@spark.apache.org> > Envoyé: Vendredi 2 Octobre 2015 18:21:35 > Objet: Re: Spark Streaming over YARN > > > If your Kafka topic has 4 partitions , and if you specify 4 Receivers, > messages from each partitions are received by a dedicated recei

Re: Spark Streaming over YARN

uot;user" <user@spark.apache.org> > Envoyé: Vendredi 2 Octobre 2015 17:43:41 > Objet: Re: Spark Streaming over YARN > > > If you're using the receiver based implementation, and want more > parallelism, you have to create multiple streams and union them together. > >

Re: Spark Streaming over YARN

r a spark job failure) > > > - Mail original - > De: "Cody Koeninger" <c...@koeninger.org> > À: "Nicolas Biau" <nib...@free.fr> > Cc: "user" <user@spark.apache.org> > Envoyé: Vendredi 2 Octobre 2015 17:43:41 > Objet: Re: Spa

Spark Streaming over YARN

Hello, I have a job receiving data from kafka (4 partitions) and persisting data inside MongoDB. It works fine, but when I deploy it inside YARN cluster (4 nodes with 2 cores) only on node is receiving all the kafka partitions and only one node is processing my RDD treatment (foreach function)

Re: Spark Streaming over YARN

If you're using the receiver based implementation, and want more parallelism, you have to create multiple streams and union them together. Or use the direct stream. On Fri, Oct 2, 2015 at 10:40 AM, wrote: > Hello, > I have a job receiving data from kafka (4 partitions) and

Re: Spark Streaming over YARN

iginal - De: "Cody Koeninger" <c...@koeninger.org> À: "Nicolas Biau" <nib...@free.fr> Cc: "user" <user@spark.apache.org> Envoyé: Vendredi 2 Octobre 2015 17:43:41 Objet: Re: Spark Streaming over YARN If you're using the receiver based implemen

Re: Spark Streaming over YARN

ree.fr Cc: "Cody Koeninger" <c...@koeninger.org>, "user" <user@spark.apache.org> Envoyé: Vendredi 2 Octobre 2015 18:01:59 Objet: Re: Spark Streaming over YARN Hi, If you need to use Receiver based approach , you can try this one : https://github.com/dibbhatt/ka

Re: Spark Streaming over YARN

e.fr > Cc: "Cody Koeninger" <c...@koeninger.org>, "user" <user@spark.apache.org> > Envoyé: Vendredi 2 Octobre 2015 18:01:59 > Objet: Re: Spark Streaming over YARN > > > Hi, > > > If you need to use Receiver based approach , you can try this o

Re: Spark Streaming over YARN

De: "Cody Koeninger" <c...@koeninger.org> > À: "Nicolas Biau" <nib...@free.fr> > Cc: "user" <user@spark.apache.org> > Envoyé: Vendredi 2 Octobre 2015 18:29:09 > Objet: Re: Spark Streaming over YARN > > > Neither of those statements are tr

Re: Spark Streaming over YARN

...@free.fr> Cc: "user" <user@spark.apache.org> Envoyé: Vendredi 2 Octobre 2015 18:29:09 Objet: Re: Spark Streaming over YARN Neither of those statements are true. You need more receivers if you want more parallelism. You don't have to manage offset positioning with the direct

Re: The Processing loading of Spark streaming on YARN is not in balance

:00 Kyle Lin kylelin2...@gmail.com: Hi all My environment info Hadoop release version: HDP 2.1 Kakfa: 0.8.1.2.1.4.0 Spark: 1.1.0 My question: I ran Spark streaming program on YARN. My Spark streaming program will read data from Kafka and doing some processing. But, I found

Re: The Processing loading of Spark streaming on YARN is not in balance

/sets From: Kyle Lin kylelin2...@gmail.com To: user@spark.apache.org user@spark.apache.org Date: 2015/04/30 14:39 Subject:The Processing loading of Spark streaming on YARN is not in balance Hi all My environment info Hadoop release version: HDP 2.1 Kakfa

The Processing loading of Spark streaming on YARN is not in balance

Hi all My environment info Hadoop release version: HDP 2.1 Kakfa: 0.8.1.2.1.4.0 Spark: 1.1.0 My question: I ran Spark streaming program on YARN. My Spark streaming program will read data from Kafka and doing some processing. But, I found there is always only ONE executor under processing

Re: The Processing loading of Spark streaming on YARN is not in balance

---2015/04/30 14:39:32---Hi all My environment info From: Kyle Lin kylelin2...@gmail.com To: user@spark.apache.org user@spark.apache.org Date: 2015/04/30 14:39 Subject: The Processing loading of Spark streaming on YARN is not in balance -- Hi all My

Re: The Processing loading of Spark streaming on YARN is not in balance

My environment info Hadoop release version: HDP 2.1 Kakfa: 0.8.1.2.1.4.0 Spark: 1.1.0 My question: I ran Spark streaming program on YARN. My Spark streaming program will read data from Kafka and doing some processing. But, I found there is always only ONE executor under processing

Spark Streaming on YARN with loss of application master

I’m looking at various HA scenarios with Spark streaming. We’re currently running a Spark streaming job that is intended to be long-lived, 24/7. We see that if we kill node managers that are hosting Spark workers, new node managers assume execution of the jobs that were running on the

Re: Spark Streaming on Yarn Input from Flume

have you fixed this issue ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-on-Yarn-Input-from-Flume-tp11755p22055.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Spark streaming on Yarn

this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-on-Yarn-tp19093.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr

Re: Spark Streaming on Yarn Input from Flume

normally .But if I run this app on yarn,no matter on yarn-client or yarn-cluster. This app can not receive data from flume,and I check the net stat,I find the port which connect to flume,didn't be listened. I am not sure,if spark streaming run on Yarn,will affect connect to flume. And then I

Spark Streaming on Yarn Input from Flume

which connect to flume,didn't be listened. I am not sure,if spark streaming run on Yarn,will affect connect to flume. And then I run the FlumeEventCount example, when I run this example on Yarn, it also can not receive data from flume. And I will be very pleasure if some one can help me

Issues in opening UI when running Spark Streaming in YARN

Hi guys, Not sure if you have similar issues. Did not find relevant tickets in JIRA. When I deploy the Spark Streaming to YARN, I have following two issues: 1. The UI port is random. It is not default 4040. I have to look at the container's log to check the UI port. Is this suppose

Re: Issues in opening UI when running Spark Streaming in YARN

not find relevant tickets in JIRA. When I deploy the Spark Streaming to YARN, I have following two issues: 1. The UI port is random. It is not default 4040. I have to look at the container's log to check the UI port. Is this suppose to be this way? 2. Most of the time, the UI does not work

Re: Issues in opening UI when running Spark Streaming in YARN

2014-07-07 11:20 GMT-07:00 Yan Fang yanfang...@gmail.com: Hi guys, Not sure if you have similar issues. Did not find relevant tickets in JIRA. When I deploy the Spark Streaming to YARN, I have following two issues: 1. The UI port is random. It is not default 4040. I have to look

Re: Issues in opening UI when running Spark Streaming in YARN

yanfang...@gmail.com: Hi guys, Not sure if you have similar issues. Did not find relevant tickets in JIRA. When I deploy the Spark Streaming to YARN, I have following two issues: 1. The UI port is random. It is not default 4040. I have to look at the container's log to check the UI port

Re: Issues in opening UI when running Spark Streaming in YARN

, not the executor containers. This is because SparkUI belongs to the SparkContext, which only exists on the driver. Andrew 2014-07-07 11:20 GMT-07:00 Yan Fang yanfang...@gmail.com: Hi guys, Not sure if you have similar issues. Did not find relevant tickets in JIRA. When I deploy the Spark Streaming

Re: Issues in opening UI when running Spark Streaming in YARN

GMT-07:00 Yan Fang yanfang...@gmail.com: Hi guys, Not sure if you have similar issues. Did not find relevant tickets in JIRA. When I deploy the Spark Streaming to YARN, I have following two issues: 1. The UI port is random. It is not default 4040. I have to look at the container's log