Hi,
after starting the shark-shell via /opt/shark/shark-0.9.1/bin/shark-withinfo -skipRddReload I receive lots of output, including the exception that /bin/java cannot be executed. But it is linked to /usr/bin/java ?!?! root#>ls -al /bin/java lrwxrwxrwx 1 root root 13 15. Apr 21:45 /bin/java -> /usr/bin/java root#>/bin/java -version java version "1.7.0_51" OpenJDK Runtime Environment (rhel-2.4.4.1.el6_5-x86_64 u51-b02) OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode) Starting the shark shell: [root@hadoop-pg-5 bin]# /opt/shark/shark-0.9.1/bin/shark-withinfo -skipRddReload -hiveconf hive.root.logger=INFO,console -skipRddReload Starting the Shark Command Line Client 14/04/15 21:45:57 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore. 14/04/15 21:45:58 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore. Logging initialized using configuration in jar:file:/opt/shark/shark-0.9.1/lib_managed/jars/edu.berkeley.cs.shark/hive-common/hive-common-0.11.0-shark-0.9.1.jar!/hive-log4j.properties 14/04/15 21:45:58 INFO SessionState: Logging initialized using configuration in jar:file:/opt/shark/shark-0.9.1/lib_managed/jars/edu.berkeley.cs.shark/hive-common/hive-common-0.11.0-shark-0.9.1.jar!/hive-log4j.properties Hive history file=/tmp/root/hive_job_log_root_22574@hadoop-pg-5.cluster_201404152145_159664609.txt 14/04/15 21:45:58 INFO exec.HiveHistory: Hive history file=/tmp/root/hive_job_log_root_22574@hadoop-pg-5.cluster_201404152145_159664609.txt 14/04/15 21:45:58 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore. 14/04/15 21:45:59 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore. 14/04/15 21:46:00 INFO slf4j.Slf4jLogger: Slf4jLogger started 14/04/15 21:46:00 INFO Remoting: Starting remoting 14/04/15 21:46:00 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@hadoop-pg-5.cluster:38835] 14/04/15 21:46:00 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@hadoop-pg-5.cluster:38835] 14/04/15 21:46:00 INFO spark.SparkEnv: Registering BlockManagerMaster 5,108: [GC 262656K->26899K(1005568K), 0,0409080 secs] 14/04/15 21:46:00 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140415214600-9537 14/04/15 21:46:00 INFO storage.MemoryStore: MemoryStore started with capacity 589.2 MB. 14/04/15 21:46:00 INFO network.ConnectionManager: Bound socket to port 51889 with id = ConnectionManagerId(hadoop-pg-5.cluster,51889) 14/04/15 21:46:00 INFO storage.BlockManagerMaster: Trying to register BlockManager 14/04/15 21:46:00 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Registering block manager hadoop-pg-5.cluster:51889 with 589.2 MB RAM 14/04/15 21:46:00 INFO storage.BlockManagerMaster: Registered BlockManager 14/04/15 21:46:00 INFO spark.HttpServer: Starting HTTP Server 14/04/15 21:46:00 INFO server.Server: jetty-7.6.8.v20121106 14/04/15 21:46:00 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:59414 14/04/15 21:46:00 INFO broadcast.HttpBroadcast: Broadcast server started at http://10.147.210.5:59414 14/04/15 21:46:01 INFO spark.SparkEnv: Registering MapOutputTracker 14/04/15 21:46:01 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-cf56ada9-d950-4abc-a1c3-76fecdc4faa3 14/04/15 21:46:01 INFO spark.HttpServer: Starting HTTP Server 14/04/15 21:46:01 INFO server.Server: jetty-7.6.8.v20121106 14/04/15 21:46:01 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:45689 14/04/15 21:46:01 INFO server.Server: jetty-7.6.8.v20121106 14/04/15 21:46:01 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/storage/rdd,null} 14/04/15 21:46:01 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/storage,null} 14/04/15 21:46:01 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/stages/stage,null} 14/04/15 21:46:01 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/stages/pool,null} 14/04/15 21:46:01 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/stages,null} 14/04/15 21:46:01 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/environment,null} 14/04/15 21:46:01 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/executors,null} 14/04/15 21:46:01 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/json,null} 14/04/15 21:46:01 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/static,null} 14/04/15 21:46:01 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/,null} 14/04/15 21:46:01 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 14/04/15 21:46:01 INFO ui.SparkUI: Started Spark Web UI at http://hadoop-pg-5.cluster:4040 14/04/15 21:46:01 INFO client.AppClient$ClientActor: Connecting to master spark://hadoop-pg-5.cluster:7077... 14/04/15 21:46:01 WARN shark.SharkEnv: Hive Hadoop shims detected local mode, but Shark is not running locally. 14/04/15 21:46:01 WARN shark.SharkEnv: Setting mapred.job.tracker to 'Spark_1397591161823' (was 'local') 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20140415214602-0010 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: app-20140415214602-0010/0 on worker-20140414105148-hadoop-pg-8.cluster-7078 (hadoop-pg-8.cluster:7078) with 2 cores 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140415214602-0010/0 on hostPort hadoop-pg-8.cluster:7078 with 2 cores, 2.0 GB RAM 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: app-20140415214602-0010/1 on worker-20140414105202-hadoop-pg-9.cluster-7078 (hadoop-pg-9.cluster:7078) with 2 cores 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140415214602-0010/1 on hostPort hadoop-pg-9.cluster:7078 with 2 cores, 2.0 GB RAM 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/1 is now RUNNING 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/0 is now RUNNING 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/1 is now FAILED (class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/1"): error=2, No such file or directory) 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor app-20140415214602-0010/1 removed: class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/1"): error=2, No such file or directory 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: app-20140415214602-0010/2 on worker-20140414105202-hadoop-pg-9.cluster-7078 (hadoop-pg-9.cluster:7078) with 2 cores 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140415214602-0010/2 on hostPort hadoop-pg-9.cluster:7078 with 2 cores, 2.0 GB RAM 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/0 is now FAILED (class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/0"): error=2, No such file or directory) 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor app-20140415214602-0010/0 removed: class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/0"): error=2, No such file or directory 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: app-20140415214602-0010/3 on worker-20140414105148-hadoop-pg-8.cluster-7078 (hadoop-pg-8.cluster:7078) with 2 cores 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140415214602-0010/3 on hostPort hadoop-pg-8.cluster:7078 with 2 cores, 2.0 GB RAM 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/3 is now RUNNING 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/2 is now RUNNING 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/3 is now FAILED (class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/3"): error=2, No such file or directory) 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor app-20140415214602-0010/3 removed: class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/3"): error=2, No such file or directory 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: app-20140415214602-0010/4 on worker-20140414105148-hadoop-pg-8.cluster-7078 (hadoop-pg-8.cluster:7078) with 2 cores 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140415214602-0010/4 on hostPort hadoop-pg-8.cluster:7078 with 2 cores, 2.0 GB RAM 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/2 is now FAILED (class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/2"): error=2, No such file or directory) 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor app-20140415214602-0010/2 removed: class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/2"): error=2, No such file or directory 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: app-20140415214602-0010/5 on worker-20140414105202-hadoop-pg-9.cluster-7078 (hadoop-pg-9.cluster:7078) with 2 cores 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140415214602-0010/5 on hostPort hadoop-pg-9.cluster:7078 with 2 cores, 2.0 GB RAM 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/4 is now RUNNING 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/5 is now RUNNING 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/4 is now FAILED (class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/4"): error=2, No such file or directory) 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor app-20140415214602-0010/4 removed: class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/4"): error=2, No such file or directory 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: app-20140415214602-0010/6 on worker-20140414105148-hadoop-pg-8.cluster-7078 (hadoop-pg-8.cluster:7078) with 2 cores 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140415214602-0010/6 on hostPort hadoop-pg-8.cluster:7078 with 2 cores, 2.0 GB RAM 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/5 is now FAILED (class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/5"): error=2, No such file or directory) 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor app-20140415214602-0010/5 removed: class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/5"): error=2, No such file or directory 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: app-20140415214602-0010/7 on worker-20140414105202-hadoop-pg-9.cluster-7078 (hadoop-pg-9.cluster:7078) with 2 cores 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140415214602-0010/7 on hostPort hadoop-pg-9.cluster:7078 with 2 cores, 2.0 GB RAM 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/7 is now RUNNING 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/6 is now RUNNING 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/6 is now FAILED (class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/6"): error=2, No such file or directory) 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor app-20140415214602-0010/6 removed: class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/6"): error=2, No such file or directory 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: app-20140415214602-0010/8 on worker-20140414105148-hadoop-pg-8.cluster-7078 (hadoop-pg-8.cluster:7078) with 2 cores 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140415214602-0010/8 on hostPort hadoop-pg-8.cluster:7078 with 2 cores, 2.0 GB RAM 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/7 is now FAILED (class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/7"): error=2, No such file or directory) 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor app-20140415214602-0010/7 removed: class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/7"): error=2, No such file or directory 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: app-20140415214602-0010/9 on worker-20140414105202-hadoop-pg-9.cluster-7078 (hadoop-pg-9.cluster:7078) with 2 cores 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140415214602-0010/9 on hostPort hadoop-pg-9.cluster:7078 with 2 cores, 2.0 GB RAM 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/8 is now RUNNING 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/9 is now RUNNING 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/8 is now FAILED (class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/8"): error=2, No such file or directory) 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor app-20140415214602-0010/8 removed: class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/8"): error=2, No such file or directory 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: app-20140415214602-0010/10 on worker-20140414105148-hadoop-pg-8.cluster-7078 (hadoop-pg-8.cluster:7078) with 2 cores 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140415214602-0010/10 on hostPort hadoop-pg-8.cluster:7078 with 2 cores, 2.0 GB RAM 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: app-20140415214602-0010/9 is now FAILED (class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/9"): error=2, No such file or directory) 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor app-20140415214602-0010/9 removed: class java.io.IOException: Cannot run program "/bin/java" (in directory "/var/run/spark/work/app-20140415214602-0010/9"): error=2, No such file or directory 14/04/15 21:46:02 ERROR client.AppClient$ClientActor: Master removed our application: FAILED; stopping client 14/04/15 21:46:02 WARN cluster.SparkDeploySchedulerBackend: Disconnected from Spark cluster! Waiting for reconnection... I do not understand that java error, any hints ?!?! br, Gerd using CM,CDH5 incl. Spark parcel
