Finally I fix it . It just cause by "ClassNotFoundException: org.apache.htrace.Trace". I can't see this message in logs on driver node,but can be found on worker node. And I modify "spark.executor.extraClassPath" in spark-default.conf still not work.Also modify classpath.txt on every node. It can work use spark-submit options "--driver-java-options -Dspark.executor.extraClassPath=/opt/cloudera/parcels/CDH/jars/htrace-core-3.1.0-incubating.jar"
Thanks. At 2015-10-09 21:10:22, "Ted Yu" <yuzhih...@gmail.com> wrote: Can you pastebin log snippet showing hbase related errors ? Please also consider posting the question on vendor's forum. On Thu, Oct 8, 2015 at 10:17 PM, roywang1024 <roywang1...@163.com> wrote: I add hbase-conf-dir to spark/conf/classpath.txt,but still error. At 2015-10-09 11:04:35, "Ted Yu" <yuzhih...@gmail.com> wrote: The second code snippet is similar to: examples//src/main/scala/org/apache/spark/examples/HBaseTest.scala See the comment in HBaseTest.scala : // please ensure HBASE_CONF_DIR is on classpath of spark driver // e.g: set it through spark.driver.extraClassPath property // in spark-defaults.conf or through --driver-class-path // command line option of spark-submit If during execution of TableInputFormatBase#initializeTable(), there was exception, table field might not have been initialized. FYI On Thu, Oct 8, 2015 at 7:54 PM, roywang1024 <roywang1...@163.com> wrote: I have try this SparkConf sparkConf = new SparkConf().setAppName("HBaseIntoSpark"); JavaSparkContext sc = new JavaSparkContext(sparkConf); Configuration conf = HBaseConfiguration.create(); Scan scan = new Scan(); scan.addFamily(Bytes.toBytes("InnerCode")); scan.addColumn(Bytes.toBytes("InnerCode"), Bytes.toBytes("")); conf.set(TableInputFormat.INPUT_TABLE, "SecuMain"); conf.set(TableInputFormat.SCAN, convertScanToString(scan)); and this SparkConf sparkConf = new SparkConf().setAppName("HBaseIntoSpark"); JavaSparkContext sc = new JavaSparkContext(sparkConf); Configuration conf = HBaseConfiguration.create(); String tableName = "SecuMain"; conf.set(TableInputFormat.INPUT_TABLE, tableName); also can't wok! Should I add hbase-site.xml to conf? Thanks. At 2015-10-09 10:35:16, "Ted Yu" <yuzhih...@gmail.com> wrote: One possibility was that hbase config, including hbase.zookeeper.quorum, was not passed to your job. hbase-site.xml should be on the classpath. Can you show snippet of your code ? Looks like you were running against hbase 1.x Cheers On Thu, Oct 8, 2015 at 7:29 PM, Roy Wang <roywang1...@163.com> wrote: I want to load hbase table into spark. JavaPairRDD<ImmutableBytesWritable, Result> hBaseRDD = sc.newAPIHadoopRDD(conf, TableInputFormat.class, ImmutableBytesWritable.class, Result.class); *when call hBaseRDD.count(),got error.* Caused by: java.lang.IllegalStateException: The input format instance has not been properly initialized. Ensure you call initializeTable either in your constructor or initialize method at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getTable(TableInputFormatBase.java:389) at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:158) ... 11 more *But when job start,I can get these logs* 2015-10-09 09:17:00[main] WARN TableInputFormatBase:447 - initializeTable called multiple times. Overwriting connection and table reference; TableInputFormatBase will not close these old references when done. Does anyone know how does this happen? Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Error-in-load-hbase-on-spark-tp24986.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org