Hello Ted, Some progress, now it seems that the spark job does get submitted,
In the spark web UI, I do see this under finished drivers. However, it seems
to not go past this step, JavaPairReceiverInputDStream<String, String>
messages = KafkaUtils.createStream(jsc, "localhost:2181", "aa", topicMap); I
do have several messages in the Kafka topic. When I run the consumer with
from-beginning option, I do see all the messages.
krishs-mbp:bin hadoop$ ./spark-submit --jars
/Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar --class
KafkaMain --master spark://krishs-mbp:7077 --deploy-mode cluster
--num-executors 100 --driver-memory 1g --executor-memory 1g --supervise
file:///Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
Sending launch command to spark://krishs-mbp:7077
Driver successfully submitted as driver-20150303085115-0007
... waiting before polling master for driver state
... polling master for driver state
State of driver-20150303085115-0007 is FINISHED
On Monday, March 2, 2015 4:31 PM, Ted Yu <[email protected]> wrote:
In AkkaUtils.scala:
val akkaLogLifecycleEvents =
conf.getBoolean("spark.akka.logLifecycleEvents", false)
Can you turn on life cycle event logging to see if you would get some more clue
?
Cheers
On Mon, Mar 2, 2015 at 3:56 PM, Krishnanand Khambadkone
<[email protected]> wrote:
I see these messages now,
spark.master -> spark://krishs-mbp:7077
Classpath elements:
Sending launch command to spark://krishs-mbp:7077
Driver successfully submitted as driver-20150302155433-0000
... waiting before polling master for driver state
... polling master for driver state
State of driver-20150302155433-0000 is FAILED
On Monday, March 2, 2015 2:42 PM, Ted Yu <[email protected]> wrote:
bq. Cause was: akka.remote.InvalidAssociation: Invalid address:
akka.tcp://sparkMaster@localhost:7077
There should be some more output following the above line.
Can you post them ?
Cheers
On Mon, Mar 2, 2015 at 2:06 PM, Krishnanand Khambadkone
<[email protected]> wrote:
Hi, I am running spark on my mac. It is reading from a kafka topic and then
writes the data to a hbase table. When I do a spark submit, I get this error,
Error connecting to master spark://localhost:7077
(akka.tcp://sparkMaster@localhost:7077), exiting.
Cause was: akka.remote.InvalidAssociation: Invalid address:
akka.tcp://sparkMaster@localhost:7077
My submit statement looks like this,
./spark-submit --jars
/Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar --class
KafkaMain --master spark://localhost:7077 --deploy-mode cluster --num-executors
100 --driver-memory 1g --executor-memory 1g
/Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar