Re: error when running spark from oozie launcher

It sounds like a mismatch in the spark version ship in oozie and the runtime one. Regards JB On Aug 18, 2016, 07:36, at 07:36, tkg_cangkul wrote: >hi olivier, thx for your reply. > >this is the full stacktrace : > >Failing Oozie Launcher, Main class >[org.apache.oozie.action.hadoop.SparkMain

Re: Aggregations with scala pairs

Agreed. Regards JB On Aug 18, 2016, 07:32, at 07:32, Olivier Girardot wrote: >CC'ing dev list, you should open a Jira and a PR related to it to >discuss it c.f. >https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-ContributingCodeChanges > > > > > >On W

Re: Spark Streaming UI duration numbers mismatch

Please find attached screenshots of the same. IMO processing time on batch page at top should be sum of durations of all jobs and similarly the duration of a job reported on batch page should be sum of durations of stages of that job. -- Thanks Jatin -- Jean-

Re: Apache Beam Spark runner

eck it out! The Apache Beam (http://beam.incubator.apache.org/) project is a unified model for building data pipelines using Google’s Dataflow programming model, and now it supports Spark as well! Take it for a ride on your Spark cluster! Thanks, Amit -- Jean-Baptiste Onofré jbono...@apach

Re: Double Counting When Using Accumulators with Spark Streaming

udera/testDirJan4/test1.text");} System.*/out/*.println(" &&&&&&&&&&&&&&&&&&&&& COUNT OF ACCUMULATOR IS "+ *accum.value(*)); *return**null*;} }); jssc.start(); If I comment rdd.saveAsTextFile I get correct count

Re: Is Spark 1.6 released?

in github as a stable release? Thanks Jung -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional comman

Re: Is Spark 1.6 released?

Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: Download Problem with Spark 1.5.2 pre-built for Hadoop 1.X

-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user

Re: Spark Submit - java.lang.IllegalArgumentException: requirement failed

ark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.o

Re: Compiling ERROR for Spark MetricsSystem

Method.java:606) at com.martiansoftware.nailgun.NGSession.run(NGSession.java:319) The IDE I'm using is Jetbrains IntelliJIDea 15 CE. I'm suspecting it's classpath issue but not sure how to further debug. Thanks Haijia -- Jean-Baptiste Onofré jbono...@apache.org http://blog.

Re: Spark streaming with Kinesis broken?

Can someone please attempt to reproduce before I open a JIRA issue for it? -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark

Re: How and where to update release notes for spark rel 1.6?

notes for spark rel 1.6? pls help. There are a few methods with changed params, and a few deprecated ones that need to be documented. Thanks Ravi -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.t

Re: How to test https://issues.apache.org/jira/browse/SPARK-10648 fix

-- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: Standalone cluster not using multiple workers for single application

. If you are not a designated recipient, you may not review, use, copy or distribute this message. If you received this in error, please notify the sender by reply e-mail and delete this message. -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Error : - No filesystem for scheme: spark

-2015 3:15 pm, "Balachandar R.A." mailto:balachandar...@gmail.com>> wrote: No.. I am not using yarn. Yarn is not running in my cluster. So, it is standalone one. Regards Bala On 02-Nov-2015 3:11 pm, "Jean-Baptiste Onofré" mailto:j...@nanthrax.net>

Re: Error : - No filesystem for scheme: spark

dar R.A. mailto:balachandar...@gmail.com>> wrote: -- Forwarded message -- From: "Balachandar R.A." mailto:balachandar...@gmail.com>> Date: 02-Nov-2015 12:53 pm Subject: Re: Error : - No filesystem for scheme: spark To: "Jean-Baptiste Onof

Re: Error : - No filesystem for scheme: spark

stemClass(FileSystem.java:2584)at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)| Any help here to move towards fixing this will be of great help Thanks Bala -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.n

Re: spark-1.5.1 application detail ui url

m the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@apache.org

RE: Spark on Yarn

Hi The compiled version (master side) and client version diverge on spark network JavaUtils. You should use the same/aligned version. RegardsJB Sent from my Samsung device Original message From: Raghuveer Chanda Date: 21/10/2015 12:33 (GMT+01:00) To: user@spark.apac

Re: Can not subscript to mailing list

Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@apac

Re: Spark 1.5 Streaming and Kinesis

Hi Phil, as you can see in the Jira, I tried with both Spark 1.5.1 and 1.6.0-SNAPSHOT, and I'm not able to reproduce your issue. What's your Java version (and I guess you use scala 2.10 for compilation, not -Pscala-2.11) ? Regards JB On 10/20/2015 03:59 PM, Jean-Baptiste Onofré

Re: Spark 1.5 Streaming and Kinesis

hil On Thu, Oct 15, 2015 at 10:35 PM, Jean-Baptiste Onofré mailto:j...@nanthrax.net>> wrote: Hi Phil, sorry I didn't have time to investigate yesterday (I was on a couple of other Apache projects ;)). I will try to do it today. I keep you posted. Regards JB

Re: Spark 1.5 Streaming and Kinesis

spark/examples Tried running the streaming job locally, and also in EMR with release 4.1.0 that includes Spark 1.5 Very strange! -- For

Re: Spark 1.5 Streaming and Kinesis

iguration that it just doesn't preserve the concrete type. So it would deserialize using the compile type instead of the runtime type. Cheers, Eugen 2015-10-15 13:41 GMT+07:00 Jean-Baptiste Onofré mailto:j...@nanthrax.net>>:

Re: Spark 1.5 Streaming and Kinesis

d running the streaming job locally, and also in EMR with release 4.1.0 that includes Spark 1.5 Very strange! ------ Forwarded message -- From: "Jean-Baptiste Onofré" mailto:j...@nanthrax.net>> To: user@spark.apache.org <mailto:user@spark.apache.org>

Re: Spark 1.5 Streaming and Kinesis

By correct, I mean: the map declaration looks good to me, so the ClassCastException is weird ;) I'm trying to reproduce the issue in order to investigate. Regards JB On 10/15/2015 08:03 AM, Jean-Baptiste Onofré wrote: Hi Phil, KinesisReceiver is part of extra. Just a dumb question: di

Re: Spark 1.5 Streaming and Kinesis

http://spark.apache.org/docs/latest/streaming-kinesis-integration.html with bin/run-example streaming.KinesisWordCountASL Am I doing something incorrect? -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talen

Re: Why is my spark executor is terminated?

5 04:42 PM, Wang, Ningjun (LNG-NPV) wrote: I checked master log before and did not find anything wrong. Unfortunately I have lost the master log now. So you think master log will tell you why executor is down? Regards, Ningjun Wang -----Original Message----- From: Jean-Baptiste Onofré [

Re: Why is my spark executor is terminated?

ning up local directories for application app-20151009201453- Thanks Ningjun Wang -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr

Re: Spark shuffle service does not work in stand alone

eader.load(DataFrameReader.scala:120) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:104) at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:219) at org.apache.saif.loaders.DataLoader$.load_json(DataLoader.scala:86) -- Jean-Baptiste Onofré jbono...@apach

Re: Why is the Spark Web GUI failing with JavaScript "Uncaught SyntaxError"?

Thanks for the update Joshua. Let me try with Spark 1.4.1. I keep you posted. Regards JB On 10/13/2015 04:17 PM, Joshua Fox wrote: * Spark 1.4.1, part of EMR emr-4.0.0 * Chrome Version 41.0.2272.118 (64-bit) on Ubuntu On Tue, Oct 13, 2015 at 3:27 PM, Jean-Baptiste Onofré mailto:j

Re: Why is the Spark Web GUI failing with JavaScript "Uncaught SyntaxError"?

er. So, it seems that my browser proxy is not the cause of this problem. Joshua -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr..

Re: Does Spark use more memory than MapReduce?

ling list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthr

Re: Build Failure

] On Thu, Oct 8, 2015 at 1:30 PM, Jean-Baptiste Onofré mailto:j...@nanthrax.net>> wrote: Hi, I just tried and it works for me (I don't have any Maven mirror on my subnet). Can you try again ? Maybe it was a temporary issue to access to Maven central. T

Re: Build Failure

: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: Running Spark in Yarn-client mode

unable to connect back to the driver. How do I resolve this? Also, I need help in locating the driver and executor node logs. Thanks. Regards, Sushrut Ikhar https://about.me/sushrutikhar <https://about.me/sushrutikhar?promo=email_sig> -- Jean-Baptiste Onofré jbono...@apache.or

Re: Help needed to reproduce bug

p24965.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste O

Re: DStream Transformation to save JSON in Cassandra 2.1

nformation transmitted by this email including damage from virus." --------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@

Re: OutOfMemoryError

11 machines (9 - 64 GB memory and 2 - 32 GB memory ). my input data of size 128 GB. How to solve this exception ? is it depends on driver.memory and execuitor.memory setting ? *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> -- Jean-Baptiste Onofré jbono...@apache.org http://b