Re: Stackoverflowerror in scala.collection

I’ve seen this when I specified “too many” where clauses in the SQL query. I was able to adjust my query to use a single ‘in’ clause rather than many ‘=’ clauses but I realize that may not be an option in all cases. Jeff On 5/4/16, 2:04 PM, "BenD" wrote: >I am

StackOverflowError in Spark SQL

I’m running Spark 1.6.0 in a standalone cluster. Periodically I’ve seen StackOverflowErrors when running queries. An example below. In the past I’ve been able to avoid such situations by ensuring we don’t have too many arguments in ‘in’ clauses or too many unioned queries both of which seem to

Timeout connecting between workers after upgrade to 1.6

I upgraded our Spark standalone cluster from 1.4.1 to 1.6.0 yesterday. We are now seeing regular timeouts between two of the workers when making connections. These workers and the same driver code worked fine running on 1.4.1 and finished in under a second. Any thoughts on what might have

Re: Standalone cluster not using multiple workers for single application

use all >workers). > >Regards >JB > >On 11/02/2015 08:56 PM, Jeff Jones wrote: >> I’ve got an a series of applications using a single standalone Spark >> cluster (v1.4.1). The cluster has 1 master and 4 workers (4 CPUs per >> worker node). I am using the start-slav

Standalone cluster not using multiple workers for single application

I’ve got an a series of applications using a single standalone Spark cluster (v1.4.1). The cluster has 1 master and 4 workers (4 CPUs per worker node). I am using the start-slave.sh script to launch the worker process on each node and I can see the nodes were successfully registered using the

Re: How can I disable logging when running local[*]?

ty, because it's a bit verbose log4j.logger.org.eclipse.jetty=WARN spark.log.threshold=OFF spark.root.logger=OFF,DRFA From: Alex Kozlov Date: Tuesday, October 6, 2015 at 10:50 AM To: Jeff Jones Cc: "user@spark.apache.org<mailto:user@spark.apache.org>" Subject: Re: How can

Re: How can I disable logging when running local[*]?

:34 PM To: Jeff Jones Cc: "user@spark.apache.org<mailto:user@spark.apache.org>" Subject: Re: How can I disable logging when running local[*]? Did you try “--driver-java-options '-Dlog4j.configuration=file:/'” and setting the log4j.rootLogger=FATAL,console? On Mon, Oct 5, 2015

How can I disable logging when running local[*]?

I’ve written an application that hosts the Spark driver in-process using “local[*]”. I’ve turned off logging in my conf/log4j.properties file. I’ve also tried putting the following code prior to creating my SparkContext. These were coupled together from various posts I’ve. None of these steps

Not able to group by Scala UDF

I’m trying to perform a Spark SQL (1.5) query containing a UDF in the select and group by clauses. From what I’ve been able to find this should be supported. A few examples include https://github.com/spirom/LearningSpark/blob/master/src/main/scala/sql/UDF.scala,

Re: ClassCastException in driver program

jars list. Unfortunately the actual error got masked by the one I sent below. Jeff From: Shixiong Zhu Date: Sunday, September 6, 2015 at 9:02 AM To: Jeff Jones Cc: "user@spark.apache.org<mailto:user@spark.apache.org>" Subject: Re: ClassCastException in driver program Looks there a

ClassCastException in driver program

We are using Scala 2.11 for a driver program that is running Spark SQL queries in a standalone cluster. I’ve rebuilt Spark for Scala 2.11 using the instructions at http://spark.apache.org/docs/latest/building-spark.html. I’ve had to work through a few dependency conflict but all-in-all it

UnknownHostNameException looking up host name with 64 characters

I've got a Spark application running on a host with 64 character FQDN. When running with Spark master local[*] I get the following error. Note, the host name should be ip-10-248-0-177.us-west-2.compute.internaldna.corp.adaptivebiotech.com but the last 6 characters are missing. The same

RE: All masters are unresponsive! Giving up.

, 2015 11:22 PM To: Jeff Jones Cc: user@spark.apache.org Subject: Re: All masters are unresponsive! Giving up. There seems to be a version mismatch somewhere. You can try and find out the cause with debug serialization information. I think the jvm flag -Dsun.io.serialization.extendedDebugInfo=true