thanks Arpit, gotcha ;)
On 16 April 2014 20:08, Arpit Tak <arpit.sparku...@gmail.com> wrote: > just set your java class path properly > > export JAVA_HOME=/usr/lib/jvm/java-7-..... (somewhat like this...whatever > version you having) > > it will work.... > > Regards, > Arpit > > > On Wed, Apr 16, 2014 at 1:24 AM, ge ko <koenig....@gmail.com> wrote: > >> Hi, >> >> >> >> after starting the shark-shell >> via /opt/shark/shark-0.9.1/bin/shark-withinfo -skipRddReload I receive lots >> of output, including the exception that /bin/java cannot be executed. But >> it is linked to /usr/bin/java ?!?! >> >> >> >> root#>ls -al /bin/java >> >> lrwxrwxrwx 1 root root 13 15. Apr 21:45 /bin/java -> /usr/bin/java >> >> root#>/bin/java -version >> >> java version "1.7.0_51" >> OpenJDK Runtime Environment (rhel-2.4.4.1.el6_5-x86_64 u51-b02) >> OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode) >> >> >> >> Starting the shark shell: >> >> >> >> [root@hadoop-pg-5 bin]# /opt/shark/shark-0.9.1/bin/shark-withinfo >> -skipRddReload >> -hiveconf hive.root.logger=INFO,console -skipRddReload >> Starting the Shark Command Line Client >> 14/04/15 21:45:57 WARN conf.HiveConf: DEPRECATED: Configuration property >> hive.metastore.local no longer has any effect. Make sure to provide a valid >> value for hive.metastore.uris if you are connecting to a remote metastore. >> 14/04/15 21:45:58 WARN conf.HiveConf: DEPRECATED: Configuration property >> hive.metastore.local no longer has any effect. Make sure to provide a valid >> value for hive.metastore.uris if you are connecting to a remote metastore. >> >> Logging initialized using configuration in >> jar:file:/opt/shark/shark-0.9.1/lib_managed/jars/edu.berkeley.cs.shark/hive-common/hive-common-0.11.0-shark-0.9.1.jar!/hive-log4j.properties >> 14/04/15 21:45:58 INFO SessionState: >> Logging initialized using configuration in >> jar:file:/opt/shark/shark-0.9.1/lib_managed/jars/edu.berkeley.cs.shark/hive-common/hive-common-0.11.0-shark-0.9.1.jar!/hive-log4j.properties >> Hive history >> file=/tmp/root/hive_job_log_root_22574@hadoop-pg-5.cluster_201404152145_159664609.txt >> 14/04/15 21:45:58 INFO exec.HiveHistory: Hive history >> file=/tmp/root/hive_job_log_root_22574@hadoop-pg-5.cluster_201404152145_159664609.txt >> 14/04/15 21:45:58 WARN conf.HiveConf: DEPRECATED: Configuration property >> hive.metastore.local no longer has any effect. Make sure to provide a valid >> value for hive.metastore.uris if you are connecting to a remote metastore. >> 14/04/15 21:45:59 WARN conf.HiveConf: DEPRECATED: Configuration property >> hive.metastore.local no longer has any effect. Make sure to provide a valid >> value for hive.metastore.uris if you are connecting to a remote metastore. >> 14/04/15 21:46:00 INFO slf4j.Slf4jLogger: Slf4jLogger started >> 14/04/15 21:46:00 INFO Remoting: Starting remoting >> 14/04/15 21:46:00 INFO Remoting: Remoting started; listening on addresses >> :[akka.tcp://spark@hadoop-pg-5.cluster:38835] >> 14/04/15 21:46:00 INFO Remoting: Remoting now listens on addresses: >> [akka.tcp://spark@hadoop-pg-5.cluster:38835] >> 14/04/15 21:46:00 INFO spark.SparkEnv: Registering BlockManagerMaster >> 5,108: [GC 262656K->26899K(1005568K), 0,0409080 secs] >> 14/04/15 21:46:00 INFO storage.DiskBlockManager: Created local directory >> at /tmp/spark-local-20140415214600-9537 >> 14/04/15 21:46:00 INFO storage.MemoryStore: MemoryStore started with >> capacity 589.2 MB. >> 14/04/15 21:46:00 INFO network.ConnectionManager: Bound socket to port >> 51889 with id = ConnectionManagerId(hadoop-pg-5.cluster,51889) >> 14/04/15 21:46:00 INFO storage.BlockManagerMaster: Trying to register >> BlockManager >> 14/04/15 21:46:00 INFO storage.BlockManagerMasterActor$BlockManagerInfo: >> Registering block manager hadoop-pg-5.cluster:51889 with 589.2 MB RAM >> 14/04/15 21:46:00 INFO storage.BlockManagerMaster: Registered BlockManager >> 14/04/15 21:46:00 INFO spark.HttpServer: Starting HTTP Server >> 14/04/15 21:46:00 INFO server.Server: jetty-7.6.8.v20121106 >> 14/04/15 21:46:00 INFO server.AbstractConnector: Started >> SocketConnector@0.0.0.0:59414 >> 14/04/15 21:46:00 INFO broadcast.HttpBroadcast: Broadcast server started >> at http://10.147.210.5:59414 >> 14/04/15 21:46:01 INFO spark.SparkEnv: Registering MapOutputTracker >> 14/04/15 21:46:01 INFO spark.HttpFileServer: HTTP File server directory >> is /tmp/spark-cf56ada9-d950-4abc-a1c3-76fecdc4faa3 >> 14/04/15 21:46:01 INFO spark.HttpServer: Starting HTTP Server >> 14/04/15 21:46:01 INFO server.Server: jetty-7.6.8.v20121106 >> 14/04/15 21:46:01 INFO server.AbstractConnector: Started >> SocketConnector@0.0.0.0:45689 >> 14/04/15 21:46:01 INFO server.Server: jetty-7.6.8.v20121106 >> 14/04/15 21:46:01 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/storage/rdd,null} >> 14/04/15 21:46:01 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/storage,null} >> 14/04/15 21:46:01 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/stages/stage,null} >> 14/04/15 21:46:01 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/stages/pool,null} >> 14/04/15 21:46:01 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/stages,null} >> 14/04/15 21:46:01 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/environment,null} >> 14/04/15 21:46:01 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/executors,null} >> 14/04/15 21:46:01 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/metrics/json,null} >> 14/04/15 21:46:01 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/static,null} >> 14/04/15 21:46:01 INFO handler.ContextHandler: started >> o.e.j.s.h.ContextHandler{/,null} >> 14/04/15 21:46:01 INFO server.AbstractConnector: Started >> SelectChannelConnector@0.0.0.0:4040 >> 14/04/15 21:46:01 INFO ui.SparkUI: Started Spark Web UI at >> http://hadoop-pg-5.cluster:4040 >> 14/04/15 21:46:01 INFO client.AppClient$ClientActor: Connecting to master >> spark://hadoop-pg-5.cluster:7077... >> 14/04/15 21:46:01 WARN shark.SharkEnv: Hive Hadoop shims detected local >> mode, but Shark is not running locally. >> 14/04/15 21:46:01 WARN shark.SharkEnv: Setting mapred.job.tracker to >> 'Spark_1397591161823' (was 'local') >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Connected to >> Spark cluster with app ID app-20140415214602-0010 >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: >> app-20140415214602-0010/0 on worker-20140414105148-hadoop-pg-8.cluster-7078 >> (hadoop-pg-8.cluster:7078) with 2 cores >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted >> executor ID app-20140415214602-0010/0 on hostPort hadoop-pg-8.cluster:7078 >> with 2 cores, 2.0 GB RAM >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: >> app-20140415214602-0010/1 on worker-20140414105202-hadoop-pg-9.cluster-7078 >> (hadoop-pg-9.cluster:7078) with 2 cores >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted >> executor ID app-20140415214602-0010/1 on hostPort hadoop-pg-9.cluster:7078 >> with 2 cores, 2.0 GB RAM >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/1 is now RUNNING >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/0 is now RUNNING >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/1 is now FAILED (class java.io.IOException: Cannot >> run program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/1"): error=2, No such file or >> directory) >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor >> app-20140415214602-0010/1 removed: class java.io.IOException: Cannot run >> program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/1"): error=2, No such file or >> directory >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: >> app-20140415214602-0010/2 on worker-20140414105202-hadoop-pg-9.cluster-7078 >> (hadoop-pg-9.cluster:7078) with 2 cores >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted >> executor ID app-20140415214602-0010/2 on hostPort hadoop-pg-9.cluster:7078 >> with 2 cores, 2.0 GB RAM >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/0 is now FAILED (class java.io.IOException: Cannot >> run program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/0"): error=2, No such file or >> directory) >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor >> app-20140415214602-0010/0 removed: class java.io.IOException: Cannot run >> program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/0"): error=2, No such file or >> directory >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: >> app-20140415214602-0010/3 on worker-20140414105148-hadoop-pg-8.cluster-7078 >> (hadoop-pg-8.cluster:7078) with 2 cores >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted >> executor ID app-20140415214602-0010/3 on hostPort hadoop-pg-8.cluster:7078 >> with 2 cores, 2.0 GB RAM >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/3 is now RUNNING >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/2 is now RUNNING >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/3 is now FAILED (class java.io.IOException: Cannot >> run program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/3"): error=2, No such file or >> directory) >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor >> app-20140415214602-0010/3 removed: class java.io.IOException: Cannot run >> program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/3"): error=2, No such file or >> directory >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: >> app-20140415214602-0010/4 on worker-20140414105148-hadoop-pg-8.cluster-7078 >> (hadoop-pg-8.cluster:7078) with 2 cores >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted >> executor ID app-20140415214602-0010/4 on hostPort hadoop-pg-8.cluster:7078 >> with 2 cores, 2.0 GB RAM >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/2 is now FAILED (class java.io.IOException: Cannot >> run program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/2"): error=2, No such file or >> directory) >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor >> app-20140415214602-0010/2 removed: class java.io.IOException: Cannot run >> program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/2"): error=2, No such file or >> directory >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: >> app-20140415214602-0010/5 on worker-20140414105202-hadoop-pg-9.cluster-7078 >> (hadoop-pg-9.cluster:7078) with 2 cores >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted >> executor ID app-20140415214602-0010/5 on hostPort hadoop-pg-9.cluster:7078 >> with 2 cores, 2.0 GB RAM >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/4 is now RUNNING >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/5 is now RUNNING >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/4 is now FAILED (class java.io.IOException: Cannot >> run program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/4"): error=2, No such file or >> directory) >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor >> app-20140415214602-0010/4 removed: class java.io.IOException: Cannot run >> program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/4"): error=2, No such file or >> directory >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: >> app-20140415214602-0010/6 on worker-20140414105148-hadoop-pg-8.cluster-7078 >> (hadoop-pg-8.cluster:7078) with 2 cores >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted >> executor ID app-20140415214602-0010/6 on hostPort hadoop-pg-8.cluster:7078 >> with 2 cores, 2.0 GB RAM >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/5 is now FAILED (class java.io.IOException: Cannot >> run program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/5"): error=2, No such file or >> directory) >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor >> app-20140415214602-0010/5 removed: class java.io.IOException: Cannot run >> program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/5"): error=2, No such file or >> directory >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: >> app-20140415214602-0010/7 on worker-20140414105202-hadoop-pg-9.cluster-7078 >> (hadoop-pg-9.cluster:7078) with 2 cores >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted >> executor ID app-20140415214602-0010/7 on hostPort hadoop-pg-9.cluster:7078 >> with 2 cores, 2.0 GB RAM >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/7 is now RUNNING >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/6 is now RUNNING >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/6 is now FAILED (class java.io.IOException: Cannot >> run program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/6"): error=2, No such file or >> directory) >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor >> app-20140415214602-0010/6 removed: class java.io.IOException: Cannot run >> program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/6"): error=2, No such file or >> directory >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: >> app-20140415214602-0010/8 on worker-20140414105148-hadoop-pg-8.cluster-7078 >> (hadoop-pg-8.cluster:7078) with 2 cores >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted >> executor ID app-20140415214602-0010/8 on hostPort hadoop-pg-8.cluster:7078 >> with 2 cores, 2.0 GB RAM >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/7 is now FAILED (class java.io.IOException: Cannot >> run program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/7"): error=2, No such file or >> directory) >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor >> app-20140415214602-0010/7 removed: class java.io.IOException: Cannot run >> program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/7"): error=2, No such file or >> directory >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: >> app-20140415214602-0010/9 on worker-20140414105202-hadoop-pg-9.cluster-7078 >> (hadoop-pg-9.cluster:7078) with 2 cores >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted >> executor ID app-20140415214602-0010/9 on hostPort hadoop-pg-9.cluster:7078 >> with 2 cores, 2.0 GB RAM >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/8 is now RUNNING >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/9 is now RUNNING >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/8 is now FAILED (class java.io.IOException: Cannot >> run program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/8"): error=2, No such file or >> directory) >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor >> app-20140415214602-0010/8 removed: class java.io.IOException: Cannot run >> program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/8"): error=2, No such file or >> directory >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: >> app-20140415214602-0010/10 on >> worker-20140414105148-hadoop-pg-8.cluster-7078 (hadoop-pg-8.cluster:7078) >> with 2 cores >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted >> executor ID app-20140415214602-0010/10 on hostPort hadoop-pg-8.cluster:7078 >> with 2 cores, 2.0 GB RAM >> 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: >> app-20140415214602-0010/9 is now FAILED (class java.io.IOException: Cannot >> run program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/9"): error=2, No such file or >> directory) >> 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor >> app-20140415214602-0010/9 removed: class java.io.IOException: Cannot run >> program "/bin/java" (in directory >> "/var/run/spark/work/app-20140415214602-0010/9"): error=2, No such file or >> directory >> 14/04/15 21:46:02 ERROR client.AppClient$ClientActor: Master removed our >> application: FAILED; stopping client >> 14/04/15 21:46:02 WARN cluster.SparkDeploySchedulerBackend: Disconnected >> from Spark cluster! Waiting for reconnection... >> >> >> >> I do not understand that java error, any hints ?!?! >> >> >> >> br, Gerd >> >> >> >> using CM,CDH5 incl. Spark parcel >> > >
