driver in queued state and not started

Hi Guys, I tried to run spark on mesos cluster. However, when I tried to submit jobs via spark-submit. The driver is in "Queued state" and not started. Which should I check? Thanks, Jared, (??) Software developer Interested in open source software, big data, Linux

Two questions about running spark on mesos

Hi Guys, Two questions about running spark on mesos. 1, Does spark configuration of conf/slaves still work when running spark on mesos? According to my observations, it seemed that conf/slaves still took effect when running spark-shell. However, it doesn't take effect when deploying

Re: A question about Spark Cluster vs Local Mode

If cluster runs out of memory, it seems that the executor will be restarted by cluster manager. Jared, (韦煜） Software developer Interested in open source software, big data, Linux From: Ascot Moss Sent: Thursday, July 28, 2016 9:48:13 AM To: user @spark Subject:

yarn.exceptions.ApplicationAttemptNotFoundException when trying to shut down spark applicaiton via yarn applicaiton --kill

Hi guys, When I tried to shut down spark application via "yarn application --kill". I run the spark application in yarn cluster mode in my laptop. I found following exception in log. org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: Application attempt appattempt_14695120

Re: Is it good choice to use DAO to store results generated by spark application?

or storing data. We need to extend spark application to query and analysis data later. Thx, Jared From: Ted Yu Sent: Wednesday, July 20, 2016 10:34:15 PM To: Yu Wei Cc: ayan guha; Rabin Banerjee; user; Deepak Sharma Subject: Re: Is it good choice to use DA

Re: Is it good choice to use DAO to store results generated by spark application?

nerjee Cc: user; Yu Wei; Deepak Sharma Subject: Re: Is it good choice to use DAO to store results generated by spark application? Just as a rain check, saving data to hbase for analytics may not be the best choice. Any specific reason for not using hdfs or hive? On 20 Jul 2016 20:57,

Re: Is it good choice to use DAO to store results generated by spark application?

Sent: Wednesday, July 20, 2016 12:34:07 PM To: Yu Wei Cc: spark users Subject: Re: Is it good choice to use DAO to store results generated by spark application? I am using DAO in spark application to write the final computation to Cassandra and it performs well. What kinds of issues you foresee usin

Re: Is it good choice to use DAO to store results generated by spark application?

Jared From: Ted Yu Sent: Wednesday, July 20, 2016 12:14:17 PM To: Andrew Ehrlich Cc: Yu Wei; user@spark.apache.org Subject: Re: Is it good choice to use DAO to store results generated by spark application? hbase-spark module is in the up-coming hbase 2.0 release. Currently it is in m

Is it good choice to use DAO to store results generated by spark application?

Hi guys, I write spark application and want to store results generated by spark application to hbase. Do I need to access hbase via java api directly? Or is it better choice to use DAO similar as traditional RDBMS? I suspect that there is major performance downgrade and other negative impact

Re: Is that possible to launch spark streaming application on yarn with only one machine?

Thanks very much for your help. Finally I understood the deploy mode with your explanation after trying different approach on my development environment. Thanks again. From: Yu Wei Sent: Saturday, July 9, 2016 3:04:40 PM To: Rabin Banerjee Cc: Mich Talebzadeh

Re: Is that possible to launch spark streaming application on yarn with only one machine?

I tried to flush the information to external system in cluster mode. It works well. I suspect that in yarn cluster mode, stdout is closed. From: Rabin Banerjee Sent: Saturday, July 9, 2016 4:22:10 AM To: Yu Wei Cc: Mich Talebzadeh; Deng Ching-Mallete; user

Re: Is that possible to launch spark streaming application on yarn with only one machine?

How could I dump data into text file? Writing to HDFS or other approach? Thanks, Jared From: Rabin Banerjee Sent: Thursday, July 7, 2016 7:04:29 PM To: Yu Wei Cc: Mich Talebzadeh; user; Deng Ching-Mallete Subject: Re: Is that possible to launch spark streaming

Re: Is that possible to launch spark streaming application on yarn with only one machine?

_______ From: Rabin Banerjee Sent: Thursday, July 7, 2016 1:04 PM To: Yu Wei Cc: Mich Talebzadeh; Deng Ching-Mallete; user@spark.apache.org Subject: Re: Is that possible to launch spark streaming application on yarn with only one machine? In yarn cluster mode , Driver is running in AM ,

Re: Is that possible to launch spark streaming application on yarn with only one machine?

Launching via client deploy mode, it works again. I'm still a little confused about the behavior difference for cluster and client mode on a single machine. Thanks, Jared From: Mich Talebzadeh Sent: Wednesday, July 6, 2016 9:46:11 PM To: Yu Wei Cc:

Re: It seemed JavaDStream.print() did not work when launching via yarn on a single node

Actually Time was printed out always. Is there any better method to debug the problem? I want to update spark/mqtt code and rebuild again to debug further. Thanks, Jared From: Saisai Shao Sent: Wednesday, July 6, 2016 9:24 PM To: Yu Wei Cc: Sean Owen; Rabin

Re: It seemed JavaDStream.print() did not work when launching via yarn on a single node

How about DStream.print(). Does it invoke collect before print on driver? From: Sean Owen Sent: Wednesday, July 6, 2016 8:20:36 PM To: Rabin Banerjee Cc: Yu Wei; user@spark.apache.org Subject: Re: It seemed JavaDStream.print() did not work when launching via

It seemed JavaDStream.print() did not work when launching via yarn on a single node

Hi guys, It seemed that when launching application via yarn on single node, JavaDStream.print() did not work. However, occasionally it worked. If launch the same application in local mode, it always worked. The code is as below, SparkConf conf = new SparkConf().setAppName("Monitor&Control");

Re: Is that possible to launch spark streaming application on yarn with only one machine?

master local[4] --driver-memory 4g --executor-memory 2g --num-executors 4 target/CollAna-1.0-SNAPSHOT.jar Any advice? Thanks, Jared ________ From: Yu Wei Sent: Tuesday, July 5, 2016 4:41 PM To: Deng Ching-Mallete Cc: user@spark.apache.org Subject: Re: Is that possible to

Re: Is that possible to launch spark streaming application on yarn with only one machine?

;, it works now. Thanks again, Jared From: odeach...@gmail.com on behalf of Deng Ching-Mallete Sent: Tuesday, July 5, 2016 4:03:28 PM To: Yu Wei Cc: user@spark.apache.org Subject: Re: Is that possible to launch spark streaming application on yarn with on

Is that possible to launch spark streaming application on yarn with only one machine?

Hi guys, I set up pseudo hadoop/yarn cluster on my labtop. I wrote a simple spark streaming program as below to receive messages with MQTTUtils. conf = new SparkConf().setAppName("Monitor&Control"); jssc = new JavaStreamingContext(conf, Durations.seconds(1)); JavaReceiverInputDStream inputDS =

Is that possible to feed web request via spark application directly?

Hi, I'm learning spark recently. I have one question about spark. Is it possible to feed web requests via spark application directly? Is there any library to be used? Or do I need to write the results from spark to HDFS/HBase? Is one spark application only to be designed to implement one sin