Re: HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting

[Chang Lim]

What command did you use to connect?  Try this:

beeline>  !connect
jdbc:hive2://localhost:1?hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice


On Thu, Aug 11, 2016 at 9:23 AM, Richard M [via Apache Spark User List] <
ml-node+s1001560n27513...@n3.nabble.com> wrote:

> I am running HiveServer2 as well and when I connect with beeline I get the
> following:
>
>  org.apache.spark.sql.internal.SessionState cannot be cast to
> org.apache.spark.sql.hive.HiveSessionState
>
>
> Do you know how to resolve this?
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-spark-user-list.1001560.n3.nabble.com/
> HiveThriftServer-and-spark-sql-hive-thriftServer-singleSession-setting-
> tp27340p27513.html
> To unsubscribe from HiveThriftServer and 
> spark.sql.hive.thriftServer.singleSession
> setting, click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/HiveThriftServer-and-spark-sql-hive-thriftServer-singleSession-setting-tp27340p27565.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting

[Chang Lim]

It's an issue with the preview build.  Switched to RC5 and all is working.
Thanks to Michael Armbrust.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/HiveThriftServer-and-spark-sql-hive-thriftServer-singleSession-setting-tp27340p27379.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting

[Chang Lim]

Would appreciate if someone:
1. Can confirm if this is an issue or 
2. Share on how to get HiveThriftServer2.startWithContext working with
shared temp table.

I am using Beeline as the JDBC client to access the temp tables of the
running Spark app.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/HiveThriftServer-and-spark-sql-hive-thriftServer-singleSession-setting-tp27340p27375.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting

[Chang Lim]

Hi,

I am on Spark 2.0 Review release.  According to Spark 2.0 docs, to share
TempTable/View, I need to:
"to run the Thrift server in the old single-session mode, please set
option spark.sql.hive.thriftServer.singleSession to true."
Question: *When using HiveThriftServer2.startWithContext(), where do I set
the above setting?*  I tried the following 4 possible places to set the flag
but it does not seem to work.  What am I missing?

val spark = SparkSession.builder.master("local[2]")
   .enableHiveSupport().appName("App")
   .config("spark.sql.hive.thriftServer.singleSession", "true")  // <--- 1.
is this the correct place to set??
   .getOrCreate()

//starts Thrift Server
//spark.conf.set("spark.sql.hive.thriftServer.singleSession", true)  <-- 2.
Tried this, don't seem to work
//spark.sparkContext.getConf.set("spark.sql.hive.thriftServer.singleSession",
"true")  <-- 3. Tried this, don't seem to work
val sql = new org.apache.spark.sql.SQLContext(spark.sparkContext)
sql.setConf("spark.sql.hive.thriftServer.singleSession", "true")  // <-- 4.
Tried this, don't seem to work
HiveThriftServer2.startWithContext(sql)





--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/HiveThriftServer-and-spark-sql-hive-thriftServer-singleSession-setting-tp27340.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Spark 2.0 Streaming and Event Time

[Chang Lim]

Here is an update to my question:
=

Tathagata Das <t...@databricks.com>
Jun 9
to me

Event time is part of windowed aggregation API. See my slides -
https://www.slideshare.net/mobile/databricks/a-deep-dive-into-structured-streaming

Let me know if it helps you to find it. Keeping it short as I am on the
mobile.
==

Chang Lim <chang...@gmail.com>
Jun 9
to Tathagata 
Hi TD,

Thanks for the reply.  But I was thinking of "sorting the events by logical
time" - more like what yesterday, the Microsoft presenter introduced
"reorder" in her slide.

The "group by" is aggregation but does not help in processing events based
on event time ordering.
=
Tathagata Das
Jun 9
to me 
Aah that's something still out of scope right now.


Chang Lim <chang...@gmail.com>
Jun 9

to Tathagata 
Wonder if we can get Microsoft to contribute "reorder" back to Spark :)

Thanks for your excellent work in Spark.

===
Michael Armbrust <mich...@databricks.com>
Jun 10
to user, me 
There is no special setting for event time (though we will be adding one for
setting a watermark in 2.1 to allow us to reduce the amount of state that
needs to be kept around).  Just window/groupBy on the on the column that is
your event time.


Chang Lim <chang...@gmail.com>
Jun 10

to Michael, user 
Yes, now I realized that.  I did exchanged emails with TD on this topic. 
The Microsoft presentation at Spark summit ("reorder" function) would be a
good addition to Spark.  Would this feature be on the road map?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-2-0-Streaming-and-Event-Time-tp27120p27232.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Running of Continuous Aggregation example

[Chang Lim]

Has anyone been able to run the code in  The Future of Real-Time in Spark
Slide
24 :"Continuous Aggregation"?

Specifically, the line: stream("jdbc:mysql//..."), 

Using Spark 2.0 preview build, I am getting the error when writing to MySQL:
Exception in thread "main" java.lang.UnsupportedOperationException: Data
source jdbc does not support streamed writing
at
org.apache.spark.sql.execution.datasources.DataSource.createSink(DataSource.scala:201)

My code:
 val logsDF = sparkSession.read.format("json")
 
.stream("file:///xxx/xxx/spark-2.0.0-preview-bin-hadoop2.4/examples/src/main/resources/people.json")
val logsDS = logsDF.as[Person]
   
logsDS.groupBy("name").sum("age").write.format("jdbc").option("checkpointLocation",
"/xxx/xxx/temp").startStream("jdbc:mysql//localhost/test")
  }

Looking at the Spark DataSource.scala source code, looks like only
ParquetFileFormat is supported?  Am I missing something?  What data sources
support streamed write? Is the example code referring to 2.0 features?

Thanks in advanced for your help.

Chang




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Running-of-Continuous-Aggregation-example-tp27229.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Spark 2.0 Streaming and Event Time

[Chang Lim]

Yes, now I realized that.  I did exchanged emails with TD on this topic.
The Microsoft presentation at Spark summit ("reorder" function) would be a
good addition to Spark.  Would this feature be on the road map?

On Thu, Jun 9, 2016 at 9:56 AM, Michael Armbrust <mich...@databricks.com>
wrote:

> There is no special setting for event time (though we will be adding one
> for setting a watermark in 2.1 to allow us to reduce the amount of state
> that needs to be kept around).  Just window/groupBy on the on the column
> that is your event time.
>
> On Wed, Jun 8, 2016 at 4:12 PM, Chang Lim <chang...@gmail.com> wrote:
>
>> Hi All,
>>
>> Does Spark 2.0 Streaming [sqlContext.read.format(...).stream(...)] support
>> Event Time?  In TD's Spark Summit talk yesterday, this is listed as a 2.0
>> feature.  Of so, where is the API or how to set it?
>>
>> Thanks in advanced,
>> Chang
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-2-0-Streaming-and-Event-Time-tp27120.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>

Spark 2.0 Streaming and Event Time

[Chang Lim]

Hi All,

Does Spark 2.0 Streaming [sqlContext.read.format(...).stream(...)] support
Event Time?  In TD's Spark Summit talk yesterday, this is listed as a 2.0
feature.  Of so, where is the API or how to set it?

Thanks in advanced,
Chang



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-2-0-Streaming-and-Event-Time-tp27120.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: HQL function Rollup and Cube

[Chang Lim]

Yes, it works for me.  Make sure the Spark machine can access the hive
machine.

On Thu, Mar 26, 2015 at 6:55 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote:

 Did you manage to connect to Hive metastore from Spark SQL. I copied hive
 conf file into Spark conf folder but when i run show tables, or do select *
 from dw_bid (dw_bid is stored in Hive) it says table not found.



 On Thu, Mar 26, 2015 at 11:43 PM, Chang Lim chang...@gmail.com wrote:

 Solved.  In IDE, project settings was missing the dependent lib jars (jar
 files under spark-xx/lib). When theses jar is not set, I got class not
 found
 error about datanucleus classes (compared to an out of memory error in
 Spark
 Shell).

 In the context of Spark Shell, these dependent jars needs to be passed in
 at
 the spark-shell command line.



 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/HQL-function-Rollup-and-Cube-tp22241p22246.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org




 --
 Deepak

Re: HQL function Rollup and Cube

[Chang Lim]

Clarification on how the HQL was invoked:

  hiveContext.sql(select a, b, count(*) from t group by a, b with rollup)

Thanks,
Chang



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/HQL-function-Rollup-and-Cube-tp22241p22244.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

HQL function Rollup and Cube

[Chang Lim]

Has anyone been able to use Hive 0.13 ROLLUP and CUBE functions in Spark
1.3's Hive Context?  According to
https://issues.apache.org/jira/browse/SPARK-2663, this has been resolved in
Spark 1.3.

I created an in-memory temp table (t) and tried to execute a ROLLUP(and
CUBE) function:

 select a, b, count(*) from t group by a, b with rollup

Got the error that with rollup is an invalid function.  Am I missing
something?

Thanks,
Chang



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/HQL-function-Rollup-and-Cube-tp22241.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: HQL function Rollup and Cube

[Chang Lim]

Solved.  In IDE, project settings was missing the dependent lib jars (jar
files under spark-xx/lib). When theses jar is not set, I got class not found
error about datanucleus classes (compared to an out of memory error in Spark
Shell).

In the context of Spark Shell, these dependent jars needs to be passed in at
the spark-shell command line.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/HQL-function-Rollup-and-Cube-tp22241p22246.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: RDD to DataFrame for using ALS under org.apache.spark.ml.recommendation.ALS

[Chang Lim]

After this line:
   val sc = new SparkContext(conf) 
You need to add this line:
   import sc.implicits._  //this is used to implicitly convert an RDD to a
DataFrame.

Hope this helps



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/RDD-to-DataFrame-for-using-ALS-under-org-apache-spark-ml-recommendation-ALS-tp22083p22247.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

SparkSQL with Streaming RDD

[Chang Lim]

Would appreciate help on:
1. How to convert streaming RDD into JavaSchemaRDD
2. How to structure the driver program to do interactive SparkSQL

Using Spark 1.0 with Java.

I have steaming code that does upateStateByKey resulting in JavaPairDStream. 
I am using JavaDStream::compute(time) to get JavaRDD.  However I am getting
the runtime expection:
   ERROR at runtime: org.apache.spark.streaming.dstream.StateDStream@18dc1b2
has not been initialized 

I know the code is executed before the stream is initialized.  Does anyone
have suggestions on how the design the code so accommodate async processing?  

Code Fragment:
//Spark SQL for the N seconds interval
SparkConf sparkConf = new
SparkConf().setMaster(SPARK_MASTER).setAppName(SQLStream);
JavaSparkContext ctx = new JavaSparkContext(sparkConf);
final JavaSQLContext sqlCtx = new
org.apache.spark.sql.api.java.JavaSQLContext(ctx);

//convert JavaPairDStream to JavaDStream
JavaDStreamTuple2lt;String,TestConnection.DiscoveryRecord javaDStream =
statefullStream.toJavaDStream();

//Convert to TupleK,U to U
JavaDStreamDiscoveryRecord javaRDD = javaDStream.map(
new FunctionTuple2lt;String,TestConnection.DiscoveryRecord,
DiscoveryRecord(){
public DiscoveryRecord
call(Tuple2String,TestConnection.DiscoveryRecord eachT) {
return eachT._2;
}
}
);

//Convert JavaDStream to JavaRDD
//ERROR next line at runtime:
org.apache.spark.streaming.dstream.StateDStream@18dc1b2 has not been
initialized
JavaRDDDiscoveryRecord computedJavaRDD = javaRDD.compute(new
Time(10));

JavaSchemaRDD schemaStatefull = sqlCtx.applySchema( computedJavaRDD ,
DiscoveryRecord.class);
schemaStatefull.registerAsTable(statefull);



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-with-Streaming-RDD-tp8774.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

DStream are not processed after upgrade to Spark 1.0

[Chang Lim]

Hi All,

I've some Streaming code in Java that works on 0.9.1.  After upgrading to
1.0 (with fix to minor API changes) DStream does not seem to be executing.
The tasks got killed in 1 second by the worker.  Any idea what is causing
it?

The worker log file is not logging my debug statements.  The following
console output seems to be executing something but my debug logs and
stream.print() is not being logged.  I see my console println statement on
the context being started.  I am using Spark standalone cluster.

Any help is much appreciated.

Console Log:
===
14/06/13 22:54:07 INFO AppClient$ClientActor: Connecting to master
spark://10.206.133.73:7077...
14/06/13 22:54:07 INFO SparkDeploySchedulerBackend: Connected to Spark
cluster with app ID app-20140613225220-0001
14/06/13 22:54:07 INFO AppClient$ClientActor: Executor added:
app-20140613225220-0001/0 on worker-20140613223732-tahiti-ins.xxx.com-46311
(tahiti-ins.xxx.com:46311) with 1 cores
14/06/13 22:54:07 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140613225220-0001/0 on hostPort tahiti-ins.xxx.com:46311 with 1 cores,
512.0 MB RAM
14/06/13 22:54:07 INFO AppClient$ClientActor: Executor updated:
app-20140613225220-0001/0 is now RUNNING
Reading CSV File
SSC Context Start
14/06/13 22:54:08 INFO StateDStream: Checkpoint interval automatically set
to 1 ms
14/06/13 22:54:08 INFO StateDStream: Checkpoint interval automatically set
to 1 ms
14/06/13 22:54:08 INFO MappedDStream: Duration for remembering RDDs set to
2 ms for org.apache.spark.streaming.dstream.MappedDStream@1158876
14/06/13 22:54:08 INFO QueueInputDStream: Duration for remembering RDDs set
to 2 ms for org.apache.spark.streaming.dstream.QueueInputDStream@b2100c
14/06/13 22:54:08 INFO TransformedDStream: Duration for remembering RDDs set
to 2 ms for org.apache.spark.streaming.dstream.TransformedDStream@3ba710
14/06/13 22:54:08 INFO ShuffledDStream: Duration for remembering RDDs set to
2 ms for org.apache.spark.streaming.dstream.ShuffledDStream@550ea2
14/06/13 22:54:08 INFO MappedDStream: Duration for remembering RDDs set to
2 ms for org.apache.spark.streaming.dstream.MappedDStream@33c98c
14/06/13 22:54:08 INFO ShuffledDStream: Duration for remembering RDDs set to
2 ms for org.apache.spark.streaming.dstream.ShuffledDStream@1f50b8f
14/06/13 22:54:08 INFO MappedDStream: Duration for remembering RDDs set to
2 ms for org.apache.spark.streaming.dstream.MappedDStream@ca0145
. . .
14/06/13 22:54:08 INFO ShuffledDStream: Initialized and validated
org.apache.spark.streaming.dstream.ShuffledDStream@1f50b8f
14/06/13 22:54:08 INFO TransformedDStream: Slide time = 1 ms
14/06/13 22:54:08 INFO TransformedDStream: Storage level =
StorageLevel(false, false, false, false, 1)
14/06/13 22:54:08 INFO TransformedDStream: Checkpoint interval = null
14/06/13 22:54:08 INFO TransformedDStream: Remember duration = 2 ms
14/06/13 22:54:08 INFO TransformedDStream: Initialized and validated
org.apache.spark.streaming.dstream.TransformedDStream@3ba710
14/06/13 22:54:08 INFO StateDStream: Slide time = 1 ms
14/06/13 22:54:08 INFO StateDStream: Storage level = StorageLevel(false,
true, false, false, 1)
14/06/13 22:54:08 INFO StateDStream: Checkpoint interval = 1 ms
14/06/13 22:54:08 INFO StateDStream: Remember duration = 2 ms
14/06/13 22:54:08 INFO StateDStream: Initialized and validated
org.apache.spark.streaming.dstream.StateDStream@140bbad
14/06/13 22:54:08 INFO ForEachDStream: Slide time = 1 ms
14/06/13 22:54:08 INFO ForEachDStream: Storage level = StorageLevel(false,
false, false, false, 1)
14/06/13 22:54:08 INFO ForEachDStream: Checkpoint interval = null
14/06/13 22:54:08 INFO ForEachDStream: Remember duration = 1 ms
14/06/13 22:54:08 INFO ForEachDStream: Initialized and validated
org.apache.spark.streaming.dstream.ForEachDStream@cf3c09
14/06/13 22:54:08 INFO RecurringTimer: Started timer for JobGenerator at
time 140272525
14/06/13 22:54:08 INFO JobGenerator: Started JobGenerator at 140272525
ms
14/06/13 22:54:08 INFO JobScheduler: Started JobScheduler



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/DStream-are-not-processed-after-upgrade-to-Spark-1-0-tp7627.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark Streaming RDD to Shark table

[Chang Lim]

OK...I needed to set the JVM class.path for the worker to find the fb class:
env.put(SPARK_JAVA_OPTS,
-Djava.class.path=/home/myInc/hive-0.9.0-bin/lib/libfb303.jar);

Now I am seeing the following spark.httpBroadcast.uri error.  What am I
missing?

java.util.NoSuchElementException: spark.httpBroadcast.uri
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:151)
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:151)
at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
at scala.collection.AbstractMap.getOrElse(Map.scala:58)
at org.apache.spark.SparkConf.get(SparkConf.scala:151)
at
org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:104)
at
org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcast.scala:70)
at
org.apache.spark.broadcast.BroadcastManager.initialize(Broadcast.scala:81)
at 
org.apache.spark.broadcast.BroadcastManager.init(Broadcast.scala:68)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:175)
at org.apache.spark.executor.Executor.init(Executor.scala:110)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receive$1.applyOrElse(CoarseGrainedExecutorBackend.scala:56)
. . .
14/05/27 15:26:45 INFO CoarseGrainedExecutorBackend: Connecting to driver:
akka.tcp://sp...@clim2-dsv.myinc.ad.myinccorp.com:3694/user/CoarseGrainedScheduler
14/05/27 15:26:46 ERROR CoarseGrainedExecutorBackend: Slave registration
failed: Duplicate executor ID: 8

===
Full Stack:
===
Spark Executor Command: /usr/lib/jvm/java-7-openjdk-i386/bin/java -cp
:/home/myInc/spark-0.9.1-bin-hadoop1/conf:/home/myInc/spark-0.9.1-bin-hadoop1/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop1.0.4.jar
-Djava.library.path=/home/myInc/hive-0.9.0-bin/lib/libfb303.jar
-Djava.library.path=/home/myInc/hive-0.9.0-bin/lib/libfb303.jar -Xms512M
-Xmx512M org.apache.spark.executor.CoarseGrainedExecutorBackend
akka.tcp://sp...@clim2-dsv.myinc.ad.myinccorp.com:3694/user/CoarseGrainedScheduler
8 tahiti-ins.myInc.ad.myInccorp.com 1
akka.tcp://sparkwor...@tahiti-ins.myinc.ad.myinccorp.com:37841/user/Worker
app-20140527152556-0029


log4j:WARN No appenders could be found for logger
(akka.event.slf4j.Slf4jLogger).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
14/05/27 15:26:44 INFO CoarseGrainedExecutorBackend: Using Spark's default
log4j profile: org/apache/spark/log4j-defaults.properties
14/05/27 15:26:44 INFO WorkerWatcher: Connecting to worker
akka.tcp://sparkwor...@tahiti-ins.myinc.ad.myinccorp.com:37841/user/Worker
14/05/27 15:26:44 INFO CoarseGrainedExecutorBackend: Connecting to driver:
akka.tcp://sp...@clim2-dsv.myinc.ad.myinccorp.com:3694/user/CoarseGrainedScheduler
14/05/27 15:26:45 INFO WorkerWatcher: Successfully connected to
akka.tcp://sparkwor...@tahiti-ins.myinc.ad.myinccorp.com:37841/user/Worker
14/05/27 15:26:45 INFO CoarseGrainedExecutorBackend: Successfully registered
with driver
14/05/27 15:26:45 INFO Slf4jLogger: Slf4jLogger started
14/05/27 15:26:45 INFO Remoting: Starting remoting
14/05/27 15:26:45 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sp...@tahiti-ins.myinc.ad.myinccorp.com:43488]
14/05/27 15:26:45 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://sp...@tahiti-ins.myinc.ad.myinccorp.com:43488]
14/05/27 15:26:45 INFO SparkEnv: Connecting to BlockManagerMaster:
akka.tcp://sp...@clim2-dsv.myinc.ad.myinccorp.com:3694/user/BlockManagerMaster
14/05/27 15:26:45 INFO DiskBlockManager: Created local directory at
/tmp/spark-local-20140527152645-b13b
14/05/27 15:26:45 INFO MemoryStore: MemoryStore started with capacity 297.0
MB.
14/05/27 15:26:45 INFO ConnectionManager: Bound socket to port 55853 with id
= ConnectionManagerId(tahiti-ins.myInc.ad.myInccorp.com,55853)
14/05/27 15:26:45 INFO BlockManagerMaster: Trying to register BlockManager
14/05/27 15:26:45 INFO BlockManagerMaster: Registered BlockManager
14/05/27 15:26:45 ERROR OneForOneStrategy: spark.httpBroadcast.uri
java.util.NoSuchElementException: spark.httpBroadcast.uri
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:151)
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:151)
at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
at scala.collection.AbstractMap.getOrElse(Map.scala:58)
at org.apache.spark.SparkConf.get(SparkConf.scala:151)
at
org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:104)
at
org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcast.scala:70)
at
org.apache.spark.broadcast.BroadcastManager.initialize(Broadcast.scala:81)
at