Re: Spark job tracker.

I am trying to create a asynchronous thread using Java executor service and launching the javaSparkContext in this thread. But it is failing with exit code 0(zero). I basically want to submit spark job in one thread and continue doing something else after submitting. Any help on this? Thanks.

Re: Spark job tracker.

Is there any thing equivalent to haddop Job (org.apache.hadoop.mapreduce.Job;) in spark? Once i submit the spark job i want to concurrently read the sparkListener interface implementation methods where i can grab the job status. I am trying to find a way to wrap the spark submit object into one

Re: Spark job tracker.

I don't understand what you're trying to do. The code will use log4j under the covers. The default configuration means writing log messages to stderr. In yarn-client mode that is your terminal screen, in yarn-cluster mode that is redirected to a file by Yarn. For the executors, that will always

Re: Spark job tracker.

I fixed the error with the yarn-client mode issue which i mentioned in my earlier post. Now i want to edit the log4j.properties to filter some of the unnecessary logs. Can you let me know where can i find this properties file. -- View this message in context:

Re: Spark job tracker.

You can upload your own log4j.properties using spark-submit's --files argument. On Tue, Jul 22, 2014 at 12:45 PM, abhiguruvayya sharath.abhis...@gmail.com wrote: I fixed the error with the yarn-client mode issue which i mentioned in my earlier post. Now i want to edit the log4j.properties to

Re: Spark job tracker.

Thanks i am able to load the file now. Can i turn off specific logs using log4j.properties. I don't want to see the below logs. How can i do this. 14/07/22 14:01:24 INFO scheduler.TaskSetManager: Starting task 2.0:129 as TID 129 on executor 3: ** (NODE_LOCAL) 14/07/22 14:01:24 INFO

Re: Spark job tracker.

The spark log classes are based on the actual class names. So if you want to filter out a package's logs you need to specify the full package name (e.g. org.apache.spark.storage instead of just spark.storage). On Tue, Jul 22, 2014 at 2:07 PM, abhiguruvayya sharath.abhis...@gmail.com wrote:

Re: Spark job tracker.

Hello Marcelo Vanzin, Can you explain bit more on this? I tried using client mode but can you explain how can i use this port to write the log or output to this port?Thanks in advance! -- View this message in context:

Re: Spark job tracker.

An also i am facing one issue. If i run the program in yarn-cluster mode it works absolutely fine but if i change it to yarn-client mode i get this below error. Application application_1405471266091_0055 failed 2 times due to AM Container for appattempt_1405471266091_0055_02 exited with

Re: Spark job tracker.

Hi Mayur, Thanks so much for the explanation. It did help me. Is there a way i can log these details on the console rather than logging it. As of now once i start my application i could see this, 14/07/10 00:48:20 INFO yarn.Client: Application report from ASM: application identifier:

Re: Spark job tracker.

That output means you're running in yarn-cluster mode. So your code is running inside the ApplicationMaster and has no access to the local terminal. If you want to see the output: - try yarn-client mode, then your code will run inside the launcher process - check the RM web ui and look at the

Re: Spark job tracker.

val sem = 0 sc.addSparkListener(new SparkListener { override def onTaskStart(taskStart: SparkListenerTaskStart) { sem +=1 } }) sc = spark context Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi https://twitter.com/mayur_rustagi

Re: Spark job tracker.

Hello Mayur, How can I implement these methods mentioned below. Do u you have any clue on this pls et me know. public void onJobStart(SparkListenerJobStart arg0) { } @Override public void onStageCompleted(SparkListenerStageCompleted arg0) { }

Re: Spark job tracker.

The application server doesnt provide json api unlike the cluster interface(8080). If you are okay to patch spark, you can use our patch to create json API, or you can use sparklistener interface in your application to get that info out. Mayur Rustagi Ph: +1 (760) 203 3257

Re: Spark job tracker.

Spark displays job status information on port 4040 using JobProgressListener, any one knows how to hook into this port and read the details? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-job-tracker-tp8367p8697.html Sent from the Apache Spark User

Re: Spark job tracker.

Hello Mayur, Are you using SparkListener interface java API? I tried using it but was unsuccessful. So need few more inputs. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-job-tracker-tp8367p8438.html Sent from the Apache Spark User List mailing

Re: Spark job tracker.

I know this is a very trivial question to ask but I'm a complete new bee to this stuff so i don't have ne clue on this. Any help is much appreciated. For example if i have a class like below, and when i run this through command line i want to see progress status. some thing like, 10%

Re: Spark job tracker.

You can use SparkListener interface to track the tasks.. another is to use JSON patch (https://github.com/apache/spark/pull/882) track tasks with json api Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi https://twitter.com/mayur_rustagi On Fri, Jun 27, 2014