Hello Guru,
Thank you for your quick response.
 This is what i get when I try executing spark-shell <master ip>:port number

C:\spark-1.4.0\bin>spark-shell <master IP>:18088
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
15/07/08 11:28:35 INFO SecurityManager: Changing view acls to: Ashish Dutt
15/07/08 11:28:35 INFO SecurityManager: Changing modify acls to: Ashish Dutt
15/07/08 11:28:35 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set
(Ashish Dutt); users with modify permissions: Set(Ashish Dutt)
15/07/08 11:28:35 INFO HttpServer: Starting HTTP Server
15/07/08 11:28:35 INFO Utils: Successfully started service 'HTTP class
server' on port 52767.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.4.0
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
1.7.0_79)
Type in expressions to have them evaluated.
Type :help for more information.
15/07/08 11:28:39 INFO SparkContext: Running Spark version 1.4.0
15/07/08 11:28:39 INFO SecurityManager: Changing view acls to: Ashish Dutt
15/07/08 11:28:39 INFO SecurityManager: Changing modify acls to: Ashish Dutt
15/07/08 11:28:39 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set
(Ashish Dutt); users with modify permissions: Set(Ashish Dutt)
15/07/08 11:28:40 INFO Slf4jLogger: Slf4jLogger started
15/07/08 11:28:40 INFO Remoting: Starting remoting
15/07/08 11:28:40 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://[email protected]:52780]
15/07/08 11:28:40 INFO Utils: Successfully started service 'sparkDriver' on
port 52780.
15/07/08 11:28:40 INFO SparkEnv: Registering MapOutputTracker
15/07/08 11:28:40 INFO SparkEnv: Registering BlockManagerMaster
15/07/08 11:28:40 INFO DiskBlockManager: Created local directory at
C:\Users\Ashish Dutt\AppData\Local\Temp\spark-80c4f1fe-37de-4aef
-9063-cae29c488382\blockmgr-a967422b-05e8-4fc1-b60b-facc7dbd4414
15/07/08 11:28:40 INFO MemoryStore: MemoryStore started with capacity 265.4
MB
15/07/08 11:28:40 INFO HttpFileServer: HTTP File server directory is
C:\Users\Ashish Dutt\AppData\Local\Temp\spark-80c4f1fe-37de-4ae
f-9063-cae29c488382\httpd-928f4485-ea08-4749-a478-59708db0fefa
15/07/08 11:28:40 INFO HttpServer: Starting HTTP Server
15/07/08 11:28:40 INFO Utils: Successfully started service 'HTTP file
server' on port 52781.
15/07/08 11:28:40 INFO SparkEnv: Registering OutputCommitCoordinator
15/07/08 11:28:40 INFO Utils: Successfully started service 'SparkUI' on
port 4040.
15/07/08 11:28:40 INFO SparkUI: Started SparkUI at http://10.228.208.74:4040
15/07/08 11:28:40 INFO Executor: Starting executor ID driver on host
localhost
15/07/08 11:28:41 INFO Executor: Using REPL class URI:
http://10.228.208.74:52767
15/07/08 11:28:41 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 52800.

15/07/08 11:28:41 INFO NettyBlockTransferService: Server created on 52800
15/07/08 11:28:41 INFO BlockManagerMaster: Trying to register BlockManager
15/07/08 11:28:41 INFO BlockManagerMasterEndpoint: Registering block
manager localhost:52800 with 265.4 MB RAM, BlockManagerId(drive
r, localhost, 52800)
15/07/08 11:28:41 INFO BlockManagerMaster: Registered BlockManager

15/07/08 11:28:41 INFO SparkILoop: Created spark context..
Spark context available as sc.
15/07/08 11:28:41 INFO HiveContext: Initializing execution hive, version
0.13.1
15/07/08 11:28:42 INFO HiveMetaStore: 0: Opening raw store with
implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/07/08 11:28:42 INFO ObjectStore: ObjectStore, initialize called
15/07/08 11:28:42 INFO Persistence: Property datanucleus.cache.level2
unknown - will be ignored
15/07/08 11:28:42 INFO Persistence: Property
hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/07/08 11:28:42 WARN Connection: BoneCP specified but not present in
CLASSPATH (or one of dependencies)
15/07/08 11:28:42 WARN Connection: BoneCP specified but not present in
CLASSPATH (or one of dependencies)
15/07/08 11:28:52 INFO ObjectStore: Setting MetaStore object pin classes
with hive.metastore.cache.pinobjtypes="Table,StorageDescrip
tor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/07/08 11:28:52 INFO MetaStoreDirectSql: MySQL check failed, assuming we
are not on mysql: Lexical error at line 1, column 5.  Enc
ountered: "@" (64), after : "".
15/07/08 11:28:53 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so do
es not have its own datastore table.
15/07/08 11:28:53 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not
 have its own datastore table.
15/07/08 11:29:00 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so do
es not have its own datastore table.
15/07/08 11:29:00 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not
 have its own datastore table.
15/07/08 11:29:02 INFO ObjectStore: Initialized ObjectStore
15/07/08 11:29:02 WARN ObjectStore: Version information not found in
metastore. hive.metastore.schema.verification is not enabled so
 recording the schema version 0.13.1aa
15/07/08 11:29:03 INFO HiveMetaStore: Added admin role in metastore
15/07/08 11:29:03 INFO HiveMetaStore: Added public role in metastore
15/07/08 11:29:03 INFO HiveMetaStore: No user is added in admin role, since
config is empty
15/07/08 11:29:03 INFO SessionState: No Tez session required at this point.
hive.execution.engine=mr.
15/07/08 11:29:03 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.

scala>

And then when I checked CDH UI, I found Spark was configured to be as
History Server. I am not using YARN.. It is a stand alone cluster with 4
nodes. I want to connect my laptop to this cluster using sparkR or pyspark
Please suggest what should I do.


Thank you,
Ashish

Sincerely,
Ashish Dutt
PhD Candidate
Department of Information Systems
University of Malaya, Lembah Pantai,
50603 Kuala Lumpur, Malaysia

On Wed, Jul 8, 2015 at 12:51 PM, Guru Medasani <[email protected]> wrote:

> Hi Ashish,
>
> Are you running Spark-on-YARN on the cluster with an instance of Spark
> History server?
>
> Also if you are using Cloudera Manager and using Spark on YARN, spark on
> yarn service has a link for the history server web UI.
>
> Can you paste the command and the output you are seeing in the thread?
>
> Guru Medasani
> [email protected]
>
>
>
> > On Jul 7, 2015, at 10:42 PM, Ashish Dutt <[email protected]> wrote:
> >
> > Hi,
> > I have CDH 5.4 installed on a linux server. It has 1 cluster in which
> spark is deployed as a history server.
> > I am trying to connect my laptop to the spark history server.
> > When I run spark-shell master ip: port number I get the following output
> > How can I verify that the worker is connected to the master?
> >
> > Thanks,
> > Ashish
> >
>
>

Reply via email to