"Too Large DataFrame" shuffle Fetch Failed exception in Spark SQL (SPARK-16753) (SPARK-9862)(SPARK-5928)(TAGs - Spark SQL, Intermediate Level, Debug)

Hi All, My spark Configuration is following. spark = SparkSession.builder.master(mesos_ip) \ .config('spark.executor.cores','3')\ .config('spark.executor.memory','8g')\ .config('spark.es.scroll.size','1')\ .config('spark.network.timeout','600s')\ .config('spark.executor.heartbeatInte

Re: Exception in spark streaming + kafka direct app

This is running in YARN cluster mode. It was restarted automatically and continued fine. I was trying to see what went wrong. AFAIK there were no task failure. Nothing in executor logs. The log I gave is in driver. After some digging, I did see that there was a rebalance in kafka logs around this

Re: Exception in spark streaming + kafka direct app

Does restarting after a few minutes solves the problem? Could be a transient issue that lasts long enough for spark task-level retries to all fail. On Tue, Feb 7, 2017 at 4:34 PM, Srikanth wrote: > Hello, > > I had a spark streaming app that reads from kafka running for a few hours > after which

Exception in spark streaming + kafka direct app

Hello, I had a spark streaming app that reads from kafka running for a few hours after which it failed with error *17/02/07 20:04:10 ERROR JobScheduler: Error generating jobs for time 148649785 ms java.lang.IllegalStateException: No current assignment for partition mt_event-5 at org.

KeyManager exception in Spark 1.6.2

I was trying to enable SSL in Spark 1.6.2 and got this exception. Not sure if I'm missing something or my keystore / truststore files got bad although keytool showed that both files are fine... = *16/09/01 04:01:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform.

Total Task size exception in Spark 1.6.0 when writing a DataFrame

Hi all, Doing some simple column transformations (e.g. trimming strings) on a DataFrame using UDFs. This DataFrame is in Avro format and being loaded off HDFS. The job has about 16,000 parts/tasks. About half way through the job, then fails with a message; org.apache.spark.SparkException: Job ab

Re: Exception in Spark-sql insertIntoJDBC command

org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.

Re: Exception in Spark-sql insertIntoJDBC command

etprops) Thanks Sri -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Exception-in-Spark-sql-insertIntoJDBC-command-tp24655p25640.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Exception in Spark-sql insertIntoJDBC command

if there is any other way to do the same in 1.4.1 version. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Exception-in-Spark-sql-insertIntoJDBC-command-tp24655.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Exception in spark

Hi all, I got an exception like “org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object” when using some where condition queries. I am using 1.4.0 version spark. But its perfectly working in hive . Please refer the following query. I have appli

Re: Exception in spark

Hi Josh Please ignore the last mail stack trace. Kindly refer the exception details. {"org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: 'Sheet1.Teams"} Regards, Ravi On Wed, Aug 12, 2015 at 1:34 AM, Ravisankar Mani wrote: > Hi

Re: Exception in spark

Hi Rosan, Thanks for your response. Kindly refer the following query and stack trace. I have checked same query in hive, It works perfectly. In case i have removed "in" in where class, it works in spark SELECT If(ISNOTNULL(SUM(`Sheet1`.`Runs`)),SUM(`Sheet1`.`Runs`),0) AS `Sheet1_Runs` ,`Sheet1`.

Re: Exception in spark

Can you share a query or stack trace? More information would make this question easier to answer. On Tue, Aug 11, 2015 at 8:50 PM, Ravisankar Mani wrote: > Hi all, > > We got an exception like > “org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call > to dataType on unresolv

Exception in spark

Hi all, We got an exception like “org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object” when using some where condition queries. I am using 1.4.0 version spark. Is this exception resolved in latest spark? Regards, Ravi

Re: Weird exception in Spark job

eas on this? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Weird-exception-in-Spark-job-tp22195p22204.html > Sent from the Apache Spark User List mailing li

Re: Weird exception in Spark job

Any Ideas on this? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Weird-exception-in-Spark-job-tp22195p22204.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Weird exception in Spark job

ty.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) ... 4 more -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Weird-exception-in-Spark-job-tp22195.html Sent from the Ap

Re: Custom UDTF with Lateral View throws ClassNotFound exception in Spark SQL CLI

A workaround trick is found and put in the ticket https://issues.apache.org/jira/browse/SPARK-4854. Hope this would be useful. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Custom-UDTF-with-Lateral-View-throws-ClassNotFound-exception-in-Spark-SQL-CLI

Re: Custom UDTF with Lateral View throws ClassNotFound exception in Spark SQL CLI

List.foreach(List.scala:318) > at > scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) > at > scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) > the rest is omitted. > > Thank you.

Custom UDTF with Lateral View throws ClassNotFound exception in Spark SQL CLI

the rest is omitted. Thank you. Shenghua -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Custom-UDTF-with-Lateral-View-throws-ClassNotFound-exception-in-Spark-SQL-CLI-tp20689.html Sent from the Apache Spark User List mailing list archiv

Re: KryoSerializer exception in Spark Streaming JAVA

erClasses(Kryo kryo) { > kryo.register(ImpressionFactsValue.class); > } > > } > > change this class to public and give a try > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/KryoSerializer-exception-in-Spark-Streami

Re: KryoSerializer exception in Spark Streaming JAVA

.n3.nabble.com/KryoSerializer-exception-in-Spark-Streaming-JAVA-tp15479p20647.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional

Re: Exception in spark sql when running a group by query

ah makes sense - Thanks Michael! On Mon, Nov 17, 2014 at 6:08 PM, Michael Armbrust wrote: > You are perhaps hitting an issue that was fixed by #3248 > ? > > On Mon, Nov 17, 2014 at 9:58 AM, Sadhan Sood > wrote: > >> While testing sparkSQL, we were runn

Re: Exception in spark sql when running a group by query

You are perhaps hitting an issue that was fixed by #3248 ? On Mon, Nov 17, 2014 at 9:58 AM, Sadhan Sood wrote: > While testing sparkSQL, we were running this group by with expression > query and got an exception. The same query worked fine on hive. > >

Exception in spark sql when running a group by query

While testing sparkSQL, we were running this group by with expression query and got an exception. The same query worked fine on hive. SELECT from_unixtime(floor(xyz.whenrequestreceived/1000.0 - 25200), '/MM/dd') as pst_date, count(*) as num_xyzs FROM all_matched_abc GROUP BY

KryoSerializer exception in Spark Streaming JAVA

utor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Please help to resolve this -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/K

Re: exception in spark 1.1.0

snappy native lib was installed as part of hadoop (cdh5.1.0) the location is in /usr/lib/hadoop/lib/ -rwxr-xr-x 1 hdfs hadoop 23904 Jul 12 14:11 libsnappy.so.1.1.3 lrwxrwxrwx 1 hdfs hadoop 18 Aug 11 20:56 libsnappy.so.1 -> libsnappy.so.1.1.3 lrwxrwxrwx 1 hdfs hadoop 18 Aug 11 20:56 libsn

Re: exception in spark 1.1.0

Can you tell us how you installed native snappy ? See D.3.1.5 in: http://hbase.apache.org/book.html#snappy.compression.installation Cheers On Sat, Sep 20, 2014 at 2:12 AM, Chen Song wrote: > I have seen below exception from spark 1.1.0. Any insights on the snappy > exception? > > 14/09/18 16:4

exception in spark 1.1.0

I have seen below exception from spark 1.1.0. Any insights on the snappy exception? 14/09/18 16:45:11 ERROR executor.Executor: Exception in task 763.1 in stage 6.0 (TID 17035) java.io.IOException: PARSING_ERROR(2) at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)

Re: Null Pointer Exception in Spark Application with Yarn Client Mode

http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml > > Thanks, > Rahul Singhal > > From: Sai Prasanna > Reply-To: "user@spark.apache.org" > Date: Monday 7 April 2014 6:56 PM > To: "user@spark.apache.org" > Subje

Re: Null Pointer Exception in Spark Application with Yarn Client Mode

ansaiprasa...@gmail.com>> Reply-To: "user@spark.apache.org<mailto:user@spark.apache.org>" mailto:user@spark.apache.org>> Date: Monday 7 April 2014 6:56 PM To: "user@spark.apache.org<mailto:user@spark.apache.org>" mailto:user@spark.apache.org>> Subject: Null

Null Pointer Exception in Spark Application with Yarn Client Mode

Hi All, I wanted Spark on Yarn to up and running. I did "*SPARK_HADOOP_VERSION=2.3.0 SPARK_YARN=true ./sbt/sbt assembly*" Then i ran "*SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop2.3.0.jar SPARK_YARN_APP_JAR=examples/target/scala-2.9.3/spark-examples_2.9.3-0.8.1