Please check the Yarn AM log to see why AM is failed to start. That's the reason why using `sc` will get such complaint.
On Fri, Dec 18, 2015 at 4:25 AM, Eran Witkon <eranwit...@gmail.com> wrote: > Hi, > I am trying to install spark 1.5.2 on Apache hadoop 2.6 and Hive and yarn > > spark-env.sh > export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop > > bash_profile > #HADOOP VARIABLES START > export JAVA_HOME=/usr/lib/jvm/java-8-oracle/ > export HADOOP_INSTALL=/usr/local/hadoop > export PATH=$PATH:$HADOOP_INSTALL/bin > export PATH=$PATH:$HADOOP_INSTALL/sbin > export HADOOP_MAPRED_HOME=$HADOOP_INSTALL > export HADOOP_COMMON_HOME=$HADOOP_INSTALL > export HADOOP_HDFS_HOME=$HADOOP_INSTALL > export YARN_HOME=$HADOOP_INSTALL > export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native > export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib" > export HADOOP_USER_CLASSPATH_FIRST=true > export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop > export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop > #HADOOP VARIABLES END > > export SPARK_HOME=/usr/local/spark > export HIVE_HOME=/usr/local/hive > export PATH=$PATH:$HIVE_HOME/bin > > > When I run spark-shell > ./bin/spark-shell --master yarn-client > > Output: > 15/12/17 22:22:07 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 15/12/17 22:22:07 INFO spark.SecurityManager: Changing view acls to: hduser > 15/12/17 22:22:07 INFO spark.SecurityManager: Changing modify acls to: > hduser > 15/12/17 22:22:07 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(hduser); users with modify permissions: Set(hduser) > 15/12/17 22:22:07 INFO spark.HttpServer: Starting HTTP Server > 15/12/17 22:22:07 INFO server.Server: jetty-8.y.z-SNAPSHOT > 15/12/17 22:22:08 INFO server.AbstractConnector: Started > SocketConnector@0.0.0.0:38389 > 15/12/17 22:22:08 INFO util.Utils: Successfully started service 'HTTP > class server' on port 38389. > Welcome to > ____ __ > / __/__ ___ _____/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 1.5.2 > /_/ > > Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java > 1.8.0_66) > Type in expressions to have them evaluated. > Type :help for more information. > 15/12/17 22:22:11 WARN util.Utils: Your hostname, eranw-Lenovo-Yoga-2-Pro > resolves to a loopback address: 127.0.1.1; using 10.0.0.1 instead (on > interface wlp1s0) > 15/12/17 22:22:11 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind > to another address > 15/12/17 22:22:11 INFO spark.SparkContext: Running Spark version 1.5.2 > 15/12/17 22:22:11 INFO spark.SecurityManager: Changing view acls to: hduser > 15/12/17 22:22:11 INFO spark.SecurityManager: Changing modify acls to: > hduser > 15/12/17 22:22:11 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(hduser); users with modify permissions: Set(hduser) > 15/12/17 22:22:11 INFO slf4j.Slf4jLogger: Slf4jLogger started > 15/12/17 22:22:11 INFO Remoting: Starting remoting > 15/12/17 22:22:12 INFO util.Utils: Successfully started service > 'sparkDriver' on port 36381. > 15/12/17 22:22:12 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkDriver@10.0.0.1:36381] > 15/12/17 22:22:12 INFO spark.SparkEnv: Registering MapOutputTracker > 15/12/17 22:22:12 INFO spark.SparkEnv: Registering BlockManagerMaster > 15/12/17 22:22:12 INFO storage.DiskBlockManager: Created local directory > at /tmp/blockmgr-139fac31-5f21-4c61-9575-3110d5205f7d > 15/12/17 22:22:12 INFO storage.MemoryStore: MemoryStore started with > capacity 530.0 MB > 15/12/17 22:22:12 INFO spark.HttpFileServer: HTTP File server directory is > /tmp/spark-955ef002-a802-49c6-b440-0656861f737c/httpd-2127cbe1-97d7-40a5-a96f-75216f115f00 > 15/12/17 22:22:12 INFO spark.HttpServer: Starting HTTP Server > 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT > 15/12/17 22:22:12 INFO server.AbstractConnector: Started > SocketConnector@0.0.0.0:36760 > 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'HTTP file > server' on port 36760. > 15/12/17 22:22:12 INFO spark.SparkEnv: Registering OutputCommitCoordinator > 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT > 15/12/17 22:22:12 INFO server.AbstractConnector: Started > SelectChannelConnector@0.0.0.0:4040 > 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'SparkUI' > on port 4040. > 15/12/17 22:22:12 INFO ui.SparkUI: Started SparkUI at http://10.0.0.1:4040 > 15/12/17 22:22:12 WARN metrics.MetricsSystem: Using default name > DAGScheduler for source because spark.app.id is not set. > 15/12/17 22:22:12 INFO client.RMProxy: Connecting to ResourceManager at / > 0.0.0.0:8032 > 15/12/17 22:22:12 INFO yarn.Client: Requesting a new application from > cluster with 1 NodeManagers > 15/12/17 22:22:12 INFO yarn.Client: Verifying our application has not > requested more than the maximum memory capability of the cluster (8192 MB > per container) > 15/12/17 22:22:12 INFO yarn.Client: Will allocate AM container, with 896 > MB memory including 384 MB overhead > 15/12/17 22:22:12 INFO yarn.Client: Setting up container launch context > for our AM > 15/12/17 22:22:12 INFO yarn.Client: Setting up the launch environment for > our AM container > 15/12/17 22:22:12 INFO yarn.Client: Preparing resources for our AM > container > 15/12/17 22:22:13 INFO yarn.Client: Uploading resource > file:/usr/local/spark/lib/spark-assembly-1.5.2-hadoop2.6.0.jar -> > hdfs://localhost:54310/user/hduser/.sparkStaging/application_1450383657924_0003/spark-assembly-1.5.2-hadoop2.6.0.jar > 15/12/17 22:22:15 INFO yarn.Client: Uploading resource > file:/tmp/spark-955ef002-a802-49c6-b440-0656861f737c/__spark_conf__3605678380745697769.zip > -> > hdfs://localhost:54310/user/hduser/.sparkStaging/application_1450383657924_0003/__spark_conf__3605678380745697769.zip > 15/12/17 22:22:15 INFO spark.SecurityManager: Changing view acls to: hduser > 15/12/17 22:22:15 INFO spark.SecurityManager: Changing modify acls to: > hduser > 15/12/17 22:22:15 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(hduser); users with modify permissions: Set(hduser) > 15/12/17 22:22:15 INFO yarn.Client: Submitting application 3 to > ResourceManager > 15/12/17 22:22:15 INFO impl.YarnClientImpl: Submitted application > application_1450383657924_0003 > 15/12/17 22:22:16 INFO yarn.Client: Application report for > application_1450383657924_0003 (state: ACCEPTED) > 15/12/17 22:22:16 INFO yarn.Client: > client token: N/A > diagnostics: N/A > ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 > queue: default > start time: 1450383735512 > final status: UNDEFINED > tracking URL: > http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003/ > user: hduser > 15/12/17 22:22:17 INFO yarn.Client: Application report for > application_1450383657924_0003 (state: ACCEPTED) > 15/12/17 22:22:18 INFO yarn.Client: Application report for > application_1450383657924_0003 (state: ACCEPTED) > 15/12/17 22:22:19 INFO yarn.Client: Application report for > application_1450383657924_0003 (state: ACCEPTED) > 15/12/17 22:22:20 INFO yarn.Client: Application report for > application_1450383657924_0003 (state: ACCEPTED) > 15/12/17 22:22:21 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka.tcp:// > sparkYarnAM@10.0.0.1:34719/user/YarnAM#-313490512]) > 15/12/17 22:22:21 INFO cluster.YarnClientSchedulerBackend: Add WebUI > Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, > Map(PROXY_HOSTS -> eranw-Lenovo-Yoga-2-Pro, PROXY_URI_BASES -> > http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003), > /proxy/application_1450383657924_0003 > 15/12/17 22:22:21 INFO ui.JettyUtils: Adding filter: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter > 15/12/17 22:22:21 INFO yarn.Client: Application report for > application_1450383657924_0003 (state: RUNNING) > 15/12/17 22:22:21 INFO yarn.Client: > client token: N/A > diagnostics: N/A > ApplicationMaster host: 10.0.0.1 > ApplicationMaster RPC port: 0 > queue: default > start time: 1450383735512 > final status: UNDEFINED > tracking URL: > http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003/ > user: hduser > 15/12/17 22:22:21 INFO cluster.YarnClientSchedulerBackend: Application > application_1450383657924_0003 has started running. > 15/12/17 22:22:21 INFO util.Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33155. > 15/12/17 22:22:21 INFO netty.NettyBlockTransferService: Server created on > 33155 > 15/12/17 22:22:21 INFO storage.BlockManagerMaster: Trying to register > BlockManager > 15/12/17 22:22:21 INFO storage.BlockManagerMasterEndpoint: Registering > block manager 10.0.0.1:33155 with 530.0 MB RAM, BlockManagerId(driver, > 10.0.0.1, 33155) > 15/12/17 22:22:21 INFO storage.BlockManagerMaster: Registered BlockManager > 15/12/17 22:22:22 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster has disassociated: 10.0.0.1:34719 > 15/12/17 22:22:22 WARN remote.ReliableDeliverySupervisor: Association with > remote system [akka.tcp://sparkYarnAM@10.0.0.1:34719] has failed, address > is now gated for [5000] ms. Reason: [Disassociated] > 15/12/17 22:22:22 INFO scheduler.EventLoggingListener: Logging events to > hdfs://localhost:54310/spark-log/application_1450383657924_0003 > 15/12/17 22:22:22 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster has disassociated: 10.0.0.1:34719 > 15/12/17 22:22:25 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka.tcp:// > sparkYarnAM@10.0.0.1:39353/user/YarnAM#1861872417]) > 15/12/17 22:22:25 INFO cluster.YarnClientSchedulerBackend: Add WebUI > Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, > Map(PROXY_HOSTS -> eranw-Lenovo-Yoga-2-Pro, PROXY_URI_BASES -> > http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003), > /proxy/application_1450383657924_0003 > 15/12/17 22:22:25 INFO ui.JettyUtils: Adding filter: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter > 15/12/17 22:22:30 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster has disassociated: 10.0.0.1:39353 > 15/12/17 22:22:30 WARN remote.ReliableDeliverySupervisor: Association with > remote system [akka.tcp://sparkYarnAM@10.0.0.1:39353] has failed, address > is now gated for [5000] ms. Reason: [Disassociated] > 15/12/17 22:22:30 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster has disassociated: 10.0.0.1:39353 > 15/12/17 22:22:30 ERROR cluster.YarnClientSchedulerBackend: Yarn > application has already exited with state FINISHED! > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/metrics/json,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/stages/stage/kill,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/api,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/static,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/executors/threadDump,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/executors/json,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/executors,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/environment/json,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/environment,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/storage/rdd/json,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/storage/rdd,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/storage/json,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/storage,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/stages/pool/json,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/stages/pool,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/stages/stage/json,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/stages/stage,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/stages/json,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/stages,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/jobs/job/json,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/jobs/job,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/jobs/json,null} > 15/12/17 22:22:30 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/jobs,null} > 15/12/17 22:22:31 INFO ui.SparkUI: Stopped Spark web UI at > http://10.0.0.1:4040 > 15/12/17 22:22:31 INFO scheduler.DAGScheduler: Stopping DAGScheduler > 15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Shutting down > all executors > 15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Asking each > executor to shut down > 15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Stopped > 15/12/17 22:22:36 INFO spark.MapOutputTrackerMasterEndpoint: > MapOutputTrackerMasterEndpoint stopped! > 15/12/17 22:22:36 INFO storage.MemoryStore: MemoryStore cleared > 15/12/17 22:22:36 INFO storage.BlockManager: BlockManager stopped > 15/12/17 22:22:36 INFO storage.BlockManagerMaster: BlockManagerMaster > stopped > 15/12/17 22:22:36 INFO spark.SparkContext: Successfully stopped > SparkContext > 15/12/17 22:22:36 INFO > scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: > OutputCommitCoordinator stopped! > 15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator: > Shutting down remote daemon. > 15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator: > Remote daemon shut down; proceeding with flushing remote transports. > 15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator: > Remoting shut down. > 15/12/17 22:22:42 INFO cluster.YarnClientSchedulerBackend: > SchedulerBackend is ready for scheduling beginning after waiting > maxRegisteredResourcesWaitingTime: 30000(ms) > 15/12/17 22:22:42 INFO repl.SparkILoop: Created spark context.. > Spark context available as sc. > 15/12/17 22:22:49 INFO hive.HiveContext: Initializing execution hive, > version 1.2.1 > 15/12/17 22:22:49 INFO client.ClientWrapper: Inspected Hadoop version: > 2.6.0 > 15/12/17 22:22:49 INFO client.ClientWrapper: Loaded > org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0 > 15/12/17 22:22:49 INFO metastore.HiveMetaStore: 0: Opening raw store with > implemenation class:org.apache.hadoop.hive.metastore.ObjectStore > 15/12/17 22:22:49 INFO metastore.ObjectStore: ObjectStore, initialize > called > 15/12/17 22:22:50 INFO DataNucleus.Persistence: Property > hive.metastore.integral.jdo.pushdown unknown - will be ignored > 15/12/17 22:22:50 INFO DataNucleus.Persistence: Property > datanucleus.cache.level2 unknown - will be ignored > 15/12/17 22:22:50 WARN DataNucleus.Connection: BoneCP specified but not > present in CLASSPATH (or one of dependencies) > 15/12/17 22:22:50 WARN DataNucleus.Connection: BoneCP specified but not > present in CLASSPATH (or one of dependencies) > 15/12/17 22:22:59 INFO metastore.ObjectStore: Setting MetaStore object pin > classes with > hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" > 15/12/17 22:23:00 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > "embedded-only" so does not have its own datastore table. > 15/12/17 22:23:00 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as > "embedded-only" so does not have its own datastore table. > 15/12/17 22:23:01 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > "embedded-only" so does not have its own datastore table. > 15/12/17 22:23:01 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as > "embedded-only" so does not have its own datastore table. > 15/12/17 22:23:02 INFO metastore.MetaStoreDirectSql: Using direct SQL, > underlying DB is DERBY > 15/12/17 22:23:02 INFO metastore.ObjectStore: Initialized ObjectStore > 15/12/17 22:23:02 WARN metastore.ObjectStore: Version information not > found in metastore. hive.metastore.schema.verification is not enabled so > recording the schema version 1.2.0 > 15/12/17 22:23:02 WARN metastore.ObjectStore: Failed to get database > default, returning NoSuchObjectException > 15/12/17 22:23:02 INFO metastore.HiveMetaStore: Added admin role in > metastore > 15/12/17 22:23:02 INFO metastore.HiveMetaStore: Added public role in > metastore > 15/12/17 22:23:03 INFO metastore.HiveMetaStore: No user is added in admin > role, since config is empty > 15/12/17 22:23:03 INFO metastore.HiveMetaStore: 0: get_all_databases > 15/12/17 22:23:03 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr > cmd=get_all_databases > 15/12/17 22:23:03 INFO metastore.HiveMetaStore: 0: get_functions: > db=default pat=* > 15/12/17 22:23:03 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr > cmd=get_functions: > db=default pat=* > 15/12/17 22:23:03 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as > "embedded-only" so does not have its own datastore table. > 15/12/17 22:23:09 INFO session.SessionState: Created local directory: > /tmp/eae419f8-7dad-417b-bc3e-77c048c88337_resources > 15/12/17 22:23:09 INFO session.SessionState: Created HDFS directory: > /tmp/hive/hduser/eae419f8-7dad-417b-bc3e-77c048c88337 > 15/12/17 22:23:09 INFO session.SessionState: Created local directory: > /tmp/hduser/eae419f8-7dad-417b-bc3e-77c048c88337 > 15/12/17 22:23:09 INFO session.SessionState: Created HDFS directory: > /tmp/hive/hduser/eae419f8-7dad-417b-bc3e-77c048c88337/_tmp_space.db > 15/12/17 22:23:09 INFO hive.HiveContext: default warehouse location is > /user/hive/warehouse > 15/12/17 22:23:09 INFO hive.HiveContext: Initializing > HiveMetastoreConnection version 1.2.1 using Spark classes. > 15/12/17 22:23:09 INFO client.ClientWrapper: Inspected Hadoop version: > 2.6.0 > 15/12/17 22:23:09 INFO client.ClientWrapper: Loaded > org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0 > 15/12/17 22:23:10 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 15/12/17 22:23:10 INFO metastore.HiveMetaStore: 0: Opening raw store with > implemenation class:org.apache.hadoop.hive.metastore.ObjectStore > 15/12/17 22:23:10 INFO metastore.ObjectStore: ObjectStore, initialize > called > 15/12/17 22:23:10 INFO DataNucleus.Persistence: Property > hive.metastore.integral.jdo.pushdown unknown - will be ignored > 15/12/17 22:23:10 INFO DataNucleus.Persistence: Property > datanucleus.cache.level2 unknown - will be ignored > 15/12/17 22:23:10 WARN DataNucleus.Connection: BoneCP specified but not > present in CLASSPATH (or one of dependencies) > 15/12/17 22:23:10 WARN DataNucleus.Connection: BoneCP specified but not > present in CLASSPATH (or one of dependencies) > 15/12/17 22:23:11 INFO metastore.ObjectStore: Setting MetaStore object pin > classes with > hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" > 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > "embedded-only" so does not have its own datastore table. > 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as > "embedded-only" so does not have its own datastore table. > 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > "embedded-only" so does not have its own datastore table. > 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as > "embedded-only" so does not have its own datastore table. > 15/12/17 22:23:12 INFO DataNucleus.Query: Reading in results for query > "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used > is closing > 15/12/17 22:23:12 INFO metastore.MetaStoreDirectSql: Using direct SQL, > underlying DB is DERBY > 15/12/17 22:23:12 INFO metastore.ObjectStore: Initialized ObjectStore > 15/12/17 22:23:13 INFO metastore.HiveMetaStore: Added admin role in > metastore > 15/12/17 22:23:13 INFO metastore.HiveMetaStore: Added public role in > metastore > 15/12/17 22:23:13 INFO metastore.HiveMetaStore: No user is added in admin > role, since config is empty > 15/12/17 22:23:13 INFO metastore.HiveMetaStore: 0: get_all_databases > 15/12/17 22:23:13 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr > cmd=get_all_databases > 15/12/17 22:23:13 INFO metastore.HiveMetaStore: 0: get_functions: > db=default pat=* > 15/12/17 22:23:13 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr > cmd=get_functions: > db=default pat=* > 15/12/17 22:23:13 INFO DataNucleus.Datastore: The class > "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as > "embedded-only" so does not have its own datastore table. > 15/12/17 22:23:18 INFO session.SessionState: Created local directory: > /tmp/7f892b9d-38ef-4fab-bc44-c1afaf07f50f_resources > 15/12/17 22:23:18 INFO session.SessionState: Created HDFS directory: > /tmp/hive/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f > 15/12/17 22:23:18 INFO session.SessionState: Created local directory: > /tmp/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f > 15/12/17 22:23:18 INFO session.SessionState: Created HDFS directory: > /tmp/hive/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f/_tmp_space.db > 15/12/17 22:23:18 INFO repl.SparkILoop: Created sql context (with Hive > support).. > SQL context available as sqlContext. > > scala> > > When I am running the following code: > val data = sc.wholeTextFiles("hdfs://localhost:54310/spark-log/*") > > I am getting exception: > java.lang.IllegalStateException: Cannot call methods on a stopped > SparkContext > at org.apache.spark.SparkContext.org > $apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104) > at > org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:2063) > at > org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:2076) > at > org.apache.spark.SparkContext.wholeTextFiles$default$2(SparkContext.scala:864) > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:21) > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:26) > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:28) > at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:30) > at $iwC$$iwC$$iwC$$iwC.<init>(<console>:32) > at $iwC$$iwC$$iwC.<init>(<console>:34) > at $iwC$$iwC.<init>(<console>:36) > at $iwC.<init>(<console>:38) > at <init>(<console>:40) > at .<init>(<console>:44) > at .<clinit>(<console>) > at .<init>(<console>:7) > at .<clinit>(<console>) > at $print(<console>) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) > at > org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340) > at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) > at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) > at > org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) > at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) > at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657) > at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665) > at org.apache.spark.repl.SparkILoop.org > $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670) > at > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997) > at > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) > at > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) > at > scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) > at org.apache.spark.repl.SparkILoop.org > $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) > at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) > at org.apache.spark.repl.Main$.main(Main.scala:31) > at org.apache.spark.repl.Main.main(Main.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > *What is wrong here? how should I configure spark to use yarn and hive > correctly?* > > Eran > >
