OK...I needed to set the JVM class.path for the worker to find the fb class: env.put("SPARK_JAVA_OPTS", "-Djava.class.path=/home/myInc/hive-0.9.0-bin/lib/libfb303.jar");
Now I am seeing the following "spark.httpBroadcast.uri" error. What am I missing? java.util.NoSuchElementException: spark.httpBroadcast.uri at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:151) at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:151) at scala.collection.MapLike$class.getOrElse(MapLike.scala:128) at scala.collection.AbstractMap.getOrElse(Map.scala:58) at org.apache.spark.SparkConf.get(SparkConf.scala:151) at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:104) at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcast.scala:70) at org.apache.spark.broadcast.BroadcastManager.initialize(Broadcast.scala:81) at org.apache.spark.broadcast.BroadcastManager.<init>(Broadcast.scala:68) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:175) at org.apache.spark.executor.Executor.<init>(Executor.scala:110) at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receive$1.applyOrElse(CoarseGrainedExecutorBackend.scala:56) . . . 14/05/27 15:26:45 INFO CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://sp...@clim2-dsv.myinc.ad.myinccorp.com:3694/user/CoarseGrainedScheduler 14/05/27 15:26:46 ERROR CoarseGrainedExecutorBackend: Slave registration failed: Duplicate executor ID: 8 =========================== Full Stack: =========================== Spark Executor Command: "/usr/lib/jvm/java-7-openjdk-i386/bin/java" "-cp" ":/home/myInc/spark-0.9.1-bin-hadoop1/conf:/home/myInc/spark-0.9.1-bin-hadoop1/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop1.0.4.jar" "-Djava.library.path=/home/myInc/hive-0.9.0-bin/lib/libfb303.jar" "-Djava.library.path=/home/myInc/hive-0.9.0-bin/lib/libfb303.jar" "-Xms512M" "-Xmx512M" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "akka.tcp://sp...@clim2-dsv.myinc.ad.myinccorp.com:3694/user/CoarseGrainedScheduler" "8" "tahiti-ins.myInc.ad.myInccorp.com" "1" "akka.tcp://sparkwor...@tahiti-ins.myinc.ad.myinccorp.com:37841/user/Worker" "app-20140527152556-0029" ======================================== log4j:WARN No appenders could be found for logger (akka.event.slf4j.Slf4jLogger). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 14/05/27 15:26:44 INFO CoarseGrainedExecutorBackend: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/05/27 15:26:44 INFO WorkerWatcher: Connecting to worker akka.tcp://sparkwor...@tahiti-ins.myinc.ad.myinccorp.com:37841/user/Worker 14/05/27 15:26:44 INFO CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://sp...@clim2-dsv.myinc.ad.myinccorp.com:3694/user/CoarseGrainedScheduler 14/05/27 15:26:45 INFO WorkerWatcher: Successfully connected to akka.tcp://sparkwor...@tahiti-ins.myinc.ad.myinccorp.com:37841/user/Worker 14/05/27 15:26:45 INFO CoarseGrainedExecutorBackend: Successfully registered with driver 14/05/27 15:26:45 INFO Slf4jLogger: Slf4jLogger started 14/05/27 15:26:45 INFO Remoting: Starting remoting 14/05/27 15:26:45 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sp...@tahiti-ins.myinc.ad.myinccorp.com:43488] 14/05/27 15:26:45 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sp...@tahiti-ins.myinc.ad.myinccorp.com:43488] 14/05/27 15:26:45 INFO SparkEnv: Connecting to BlockManagerMaster: akka.tcp://sp...@clim2-dsv.myinc.ad.myinccorp.com:3694/user/BlockManagerMaster 14/05/27 15:26:45 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140527152645-b13b 14/05/27 15:26:45 INFO MemoryStore: MemoryStore started with capacity 297.0 MB. 14/05/27 15:26:45 INFO ConnectionManager: Bound socket to port 55853 with id = ConnectionManagerId(tahiti-ins.myInc.ad.myInccorp.com,55853) 14/05/27 15:26:45 INFO BlockManagerMaster: Trying to register BlockManager 14/05/27 15:26:45 INFO BlockManagerMaster: Registered BlockManager 14/05/27 15:26:45 ERROR OneForOneStrategy: spark.httpBroadcast.uri java.util.NoSuchElementException: spark.httpBroadcast.uri at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:151) at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:151) at scala.collection.MapLike$class.getOrElse(MapLike.scala:128) at scala.collection.AbstractMap.getOrElse(Map.scala:58) at org.apache.spark.SparkConf.get(SparkConf.scala:151) at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:104) at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcast.scala:70) at org.apache.spark.broadcast.BroadcastManager.initialize(Broadcast.scala:81) at org.apache.spark.broadcast.BroadcastManager.<init>(Broadcast.scala:68) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:175) at org.apache.spark.executor.Executor.<init>(Executor.scala:110) at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receive$1.applyOrElse(CoarseGrainedExecutorBackend.scala:56) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) at akka.actor.ActorCell.invoke(ActorCell.scala:456) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 14/05/27 15:26:45 INFO CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://sp...@clim2-dsv.myinc.ad.myinccorp.com:3694/user/CoarseGrainedScheduler 14/05/27 15:26:46 ERROR CoarseGrainedExecutorBackend: Slave registration failed: Duplicate executor ID: 8 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-RDD-to-Shark-table-tp5063p6485.html Sent from the Apache Spark User List mailing list archive at Nabble.com.