Hello Guru, Thank you for your quick response. This is what i get when I try executing spark-shell <master ip>:port number
C:\spark-1.4.0\bin>spark-shell <master IP>:18088 log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/07/08 11:28:35 INFO SecurityManager: Changing view acls to: Ashish Dutt 15/07/08 11:28:35 INFO SecurityManager: Changing modify acls to: Ashish Dutt 15/07/08 11:28:35 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set (Ashish Dutt); users with modify permissions: Set(Ashish Dutt) 15/07/08 11:28:35 INFO HttpServer: Starting HTTP Server 15/07/08 11:28:35 INFO Utils: Successfully started service 'HTTP class server' on port 52767. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.4.0 /_/ Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_79) Type in expressions to have them evaluated. Type :help for more information. 15/07/08 11:28:39 INFO SparkContext: Running Spark version 1.4.0 15/07/08 11:28:39 INFO SecurityManager: Changing view acls to: Ashish Dutt 15/07/08 11:28:39 INFO SecurityManager: Changing modify acls to: Ashish Dutt 15/07/08 11:28:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set (Ashish Dutt); users with modify permissions: Set(Ashish Dutt) 15/07/08 11:28:40 INFO Slf4jLogger: Slf4jLogger started 15/07/08 11:28:40 INFO Remoting: Starting remoting 15/07/08 11:28:40 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:52780] 15/07/08 11:28:40 INFO Utils: Successfully started service 'sparkDriver' on port 52780. 15/07/08 11:28:40 INFO SparkEnv: Registering MapOutputTracker 15/07/08 11:28:40 INFO SparkEnv: Registering BlockManagerMaster 15/07/08 11:28:40 INFO DiskBlockManager: Created local directory at C:\Users\Ashish Dutt\AppData\Local\Temp\spark-80c4f1fe-37de-4aef -9063-cae29c488382\blockmgr-a967422b-05e8-4fc1-b60b-facc7dbd4414 15/07/08 11:28:40 INFO MemoryStore: MemoryStore started with capacity 265.4 MB 15/07/08 11:28:40 INFO HttpFileServer: HTTP File server directory is C:\Users\Ashish Dutt\AppData\Local\Temp\spark-80c4f1fe-37de-4ae f-9063-cae29c488382\httpd-928f4485-ea08-4749-a478-59708db0fefa 15/07/08 11:28:40 INFO HttpServer: Starting HTTP Server 15/07/08 11:28:40 INFO Utils: Successfully started service 'HTTP file server' on port 52781. 15/07/08 11:28:40 INFO SparkEnv: Registering OutputCommitCoordinator 15/07/08 11:28:40 INFO Utils: Successfully started service 'SparkUI' on port 4040. 15/07/08 11:28:40 INFO SparkUI: Started SparkUI at http://10.228.208.74:4040 15/07/08 11:28:40 INFO Executor: Starting executor ID driver on host localhost 15/07/08 11:28:41 INFO Executor: Using REPL class URI: http://10.228.208.74:52767 15/07/08 11:28:41 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52800. 15/07/08 11:28:41 INFO NettyBlockTransferService: Server created on 52800 15/07/08 11:28:41 INFO BlockManagerMaster: Trying to register BlockManager 15/07/08 11:28:41 INFO BlockManagerMasterEndpoint: Registering block manager localhost:52800 with 265.4 MB RAM, BlockManagerId(drive r, localhost, 52800) 15/07/08 11:28:41 INFO BlockManagerMaster: Registered BlockManager 15/07/08 11:28:41 INFO SparkILoop: Created spark context.. Spark context available as sc. 15/07/08 11:28:41 INFO HiveContext: Initializing execution hive, version 0.13.1 15/07/08 11:28:42 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/07/08 11:28:42 INFO ObjectStore: ObjectStore, initialize called 15/07/08 11:28:42 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored 15/07/08 11:28:42 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 15/07/08 11:28:42 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/07/08 11:28:42 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/07/08 11:28:52 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescrip tor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 15/07/08 11:28:52 INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Enc ountered: "@" (64), after : "". 15/07/08 11:28:53 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so do es not have its own datastore table. 15/07/08 11:28:53 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 15/07/08 11:29:00 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so do es not have its own datastore table. 15/07/08 11:29:00 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 15/07/08 11:29:02 INFO ObjectStore: Initialized ObjectStore 15/07/08 11:29:02 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa 15/07/08 11:29:03 INFO HiveMetaStore: Added admin role in metastore 15/07/08 11:29:03 INFO HiveMetaStore: Added public role in metastore 15/07/08 11:29:03 INFO HiveMetaStore: No user is added in admin role, since config is empty 15/07/08 11:29:03 INFO SessionState: No Tez session required at this point. hive.execution.engine=mr. 15/07/08 11:29:03 INFO SparkILoop: Created sql context (with Hive support).. SQL context available as sqlContext. scala> And then when I checked CDH UI, I found Spark was configured to be as History Server. I am not using YARN.. It is a stand alone cluster with 4 nodes. I want to connect my laptop to this cluster using sparkR or pyspark Please suggest what should I do. Thank you, Ashish Sincerely, Ashish Dutt PhD Candidate Department of Information Systems University of Malaya, Lembah Pantai, 50603 Kuala Lumpur, Malaysia On Wed, Jul 8, 2015 at 12:51 PM, Guru Medasani <[email protected]> wrote: > Hi Ashish, > > Are you running Spark-on-YARN on the cluster with an instance of Spark > History server? > > Also if you are using Cloudera Manager and using Spark on YARN, spark on > yarn service has a link for the history server web UI. > > Can you paste the command and the output you are seeing in the thread? > > Guru Medasani > [email protected] > > > > > On Jul 7, 2015, at 10:42 PM, Ashish Dutt <[email protected]> wrote: > > > > Hi, > > I have CDH 5.4 installed on a linux server. It has 1 cluster in which > spark is deployed as a history server. > > I am trying to connect my laptop to the spark history server. > > When I run spark-shell master ip: port number I get the following output > > How can I verify that the worker is connected to the master? > > > > Thanks, > > Ashish > > > >
