Hi, I installed hbase-0.98.6-hadoop2. It's working not any problem with that.
When i am try to run spark hbase python examples, (wordcount examples working - not python issue) ./bin/spark-submit --master local --driver-class-path ./examples/target/spark-examples_2.10-1.1.0.jar ./examples/src/main/python/hbase_inputformat.py localhost myhbasetable the process exit with ClassNotFoundException... I search lots of blogs, sites all says "spark 1.1 version built with hbase 0.94.6 rebuild with own hbase version". I try first, change hbase version number - in pom.xml -- nothing found maven central I try second, compile hbase from src and copy hbase/lib folder hbase jars to spark/lib_managed folder and edit spark-defaults.conf my spark-defaults.conf spark.executor.extraClassPath /home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-server-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-protocol-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-hadoop2-compat-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-client-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/hbase-commont-0.98.6-hadoop2.jar:/home/downloads/spark/spark-1.1.0/lib_managed/jars/htrace-core-2.04.jar My question is how i can work with hbase 0.98.6-hadoop2 with spark 1.1.0 Here is the exception message.... Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/10/03 11:27:15 WARN Utils: Your hostname, testmachine1.tracescore.com resolves to a loopback address: 127.0.0.1; using 1.1.1.1 instead (on interface eth0) 14/10/03 11:27:15 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 14/10/03 11:27:15 INFO SecurityManager: Changing view acls to: root, 14/10/03 11:27:15 INFO SecurityManager: Changing modify acls to: root, 14/10/03 11:27:15 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, ); users with modify permissions: Set(root, ) 14/10/03 11:27:16 INFO Slf4jLogger: Slf4jLogger started 14/10/03 11:27:16 INFO Remoting: Starting remoting 14/10/03 11:27:16 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkdri...@1-1-1-1-1.rev.mydomain.io:49256] 14/10/03 11:27:16 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkdri...@1-1-1-1-1.rev.mydomain.io:49256] 14/10/03 11:27:16 INFO Utils: Successfully started service 'sparkDriver' on port 49256. 14/10/03 11:27:16 INFO SparkEnv: Registering MapOutputTracker 14/10/03 11:27:16 INFO SparkEnv: Registering BlockManagerMaster 14/10/03 11:27:16 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20141003112716-298d 14/10/03 11:27:16 INFO Utils: Successfully started service 'Connection manager for block manager' on port 35106. 14/10/03 11:27:16 INFO ConnectionManager: Bound socket to port 35106 with id = ConnectionManagerId(1-1-1-1-1.rev.mydomain.io,35106) 14/10/03 11:27:16 INFO MemoryStore: MemoryStore started with capacity 267.3 MB 14/10/03 11:27:16 INFO BlockManagerMaster: Trying to register BlockManager 14/10/03 11:27:16 INFO BlockManagerMasterActor: Registering block manager 1-1-1-1-1.rev.mydomain.io:35106 with 267.3 MB RAM 14/10/03 11:27:16 INFO BlockManagerMaster: Registered BlockManager 14/10/03 11:27:16 INFO HttpFileServer: HTTP File server directory is /tmp/spark-f60b0533-998f-4af2-a208-d04c571eab82 14/10/03 11:27:16 INFO HttpServer: Starting HTTP Server 14/10/03 11:27:16 INFO Utils: Successfully started service 'HTTP file server' on port 49611. 14/10/03 11:27:16 INFO Utils: Successfully started service 'SparkUI' on port 4040. 14/10/03 11:27:16 INFO SparkUI: Started SparkUI at http://1-1-1-1-1.rev.mydomain.io:4040 14/10/03 11:27:16 INFO Utils: Copying /home/downloads/spark/spark-1.1.0/./examples/src/main/python/hbase_inputformat.py to /tmp/spark-7232227a-0547-454e-9f68-805fa7b0c2f0/hbase_inputformat.py 14/10/03 11:27:16 INFO SparkContext: Added file file:/home/downloads/spark/spark-1.1.0/./examples/src/main/python/hbase_inputformat.py at http://1.1.1.1:49611/files/hbase_inputformat.py with timestamp 1412324836837 14/10/03 11:27:16 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkdri...@1-1-1-1-1.rev.mydomain.io:49256/user/HeartbeatReceiver Traceback (most recent call last): File "/home/downloads/spark/spark-1.1.0/./examples/src/main/python/hbase_inputformat.py", line 70, in <module> conf=conf) File "/home/downloads/spark/spark-1.1.0/python/pyspark/context.py", line 471, in newAPIHadoopRDD jconf, batchSize) File "/usr/lib/python2.6/site-packages/py4j-0.8.2.1-py2.6.egg/py4j/java_gateway.py", line 538, in __call__ self.target_id, self.name) File "/usr/lib/python2.6/site-packages/py4j-0.8.2.1-py2.6.egg/py4j/protocol.py", line 300, in get_return_value format(target_id, '.', name), value) py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD. : java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable at java.net.URLClassLoader$1.run(URLClassLoader.java:217) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:205) at java.lang.ClassLoader.loadClass(ClassLoader.java:323) at java.lang.ClassLoader.loadClass(ClassLoader.java:268) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at org.apache.spark.util.Utils$.classForName(Utils.scala:150) at org.apache.spark.api.python.PythonRDD$.newAPIHadoopRDDFromClassNames(PythonRDD.scala:451) at org.apache.spark.api.python.PythonRDD$.newAPIHadoopRDD(PythonRDD.scala:436) at org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD(PythonRDD.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:622) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) at py4j.Gateway.invoke(Gateway.java:259) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:207) at java.lang.Thread.run(Thread.java:701) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-1-0-hbase-0-98-6-hadoop2-version-py4j-protocol-Py4JJavaError-java-lang-ClassNotFoundException-tp15666.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
