Re: Spark with HBase Error - Py4JJavaError

gt; > > > *From:* Puneet Tripathi [mailto:puneet.tripa...@dunnhumby.com] > *Sent:* Thursday, July 07, 2016 12:42 PM > *To:* user@spark.apache.org > *Subject:* Spark with HBase Error - Py4JJavaError > > > > Hi, > > > > We are running Hbase in fully distributed mode.

RE: Spark with HBase Error - Py4JJavaError

Guys, Please can anyone help on the issue below? Puneet From: Puneet Tripathi [mailto:puneet.tripa...@dunnhumby.com] Sent: Thursday, July 07, 2016 12:42 PM To: user@spark.apache.org Subject: Spark with HBase Error - Py4JJavaError Hi, We are running Hbase in fully distributed mode. I tried to

Spark with HBase Error - Py4JJavaError

Hi, We are running Hbase in fully distributed mode. I tried to connect to Hbase via pyspark and then write to hbase using saveAsNewAPIHadoopDataset , but it failed the error says: Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.saveAsHadoopDataset

Re: Recommended way to push data into HBase through Spark streaming

Forgot to add, I'm on HBase 1.0.0-cdh5.4.5, so can't use HBaseContext. And spark version is 1.6.1 [image: http://] Tariq, Mohammad about.me/mti [image: http://] <http://about.me/mti> On Thu, Jun 16, 2016 at 10:12 PM, Mohammad Tariq wrote: > Hi group, > > I have a

Recommended way to push data into HBase through Spark streaming

Hi group, I have a streaming job which reads data from Kafka, performs some computation and pushes the result into HBase. Actually the results are pushed into 3 different HBase tables. So I was wondering what could be the best way to achieve this. Since each executor will open its own HBase

Re: Spark Streaming application failing with Kerboros issue while writing data to HBase

t; On Mon, Jun 13, 2016 at 4:44 AM, Kamesh wrote: > >> Hi All, >> We are building a spark streaming application and that application >> writes data to HBase table. But writes/reads are failing with following >> exception >> >> 16/06/13 04:35:16 ERROR ipc.Abst

Re: Spark Streaming application failing with Kerboros issue while writing data to HBase

cation writes > data to HBase table. But writes/reads are failing with following exception > > 16/06/13 04:35:16 ERROR ipc.AbstractRpcClient: SASL authentication failed. > The most likely cause is missing or invalid credentials. Consider 'kinit'. > > javax.security.sasl.

Spark Streaming application failing with Kerboros issue while writing data to HBase

Hi All, We are building a spark streaming application and that application writes data to HBase table. But writes/reads are failing with following exception 16/06/13 04:35:16 ERROR ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials

Re: Spark Streaming stateful operation to HBase

Hi, Check the number of records inside the DStream at a batch before you do the save. Gist the code with mapWithState and save? Jacek On 9 Jun 2016 7:58 a.m., "soumick dasgupta" wrote: Hi, I am using mapwithstate to keep the state and then ouput the result to HBase. The problem I

Spark Streaming stateful operation to HBase

Hi, I am using mapwithstate to keep the state and then ouput the result to HBase. The problem I am facing is when there are no files arriving, the RDD is still emitting the previous state result due to the checkpoint. Is there a way I can restrict not to write that result to HBase, i.e., when the

Re: Bulk loading Serialized RDD into Hbase throws KryoException - IndexOutOfBoundsException

s.foreachPartition{ itr => >> >> val hConf = HBaseConfiguration.create() >> >> hConf.setInt("hbase.client.write.buffer", 16097152) >> >> val table = new HTable(hConf, tableName) >> >> //table.setWriteBuffer

Re: Bulk loading Serialized RDD into Hbase throws KryoException - IndexOutOfBoundsException

erSize(8388608) >> >> itr.grouped(100).foreach(table.put(_)) // << Exception happens at >> this point >> >> table.close() >> >> } >> >> } >> >> >> >> I am using hbase 0.98.12 mapr dis

Re: Bulk loading Serialized RDD into Hbase throws KryoException - IndexOutOfBoundsException

Conf = HBaseConfiguration.create() > > hConf.setInt("hbase.client.write.buffer", 16097152) > > val table = new HTable(hConf, tableName) > > //table.setWriteBufferSize(8388608) > > *itr.grouped(100).foreach(table.put(_)) * // << Exception >

Re: Bulk loading Serialized RDD into Hbase throws KryoException - IndexOutOfBoundsException

le.setWriteBufferSize(8388608) > > itr.grouped(100).foreach(table.put(_)) // << Exception happens at > this point > > table.close() > > } > > } > > > > I am using hbase 0.98.12 mapr distribution. > > > > Thanks > &

Re: Bulk loading Serialized RDD into Hbase throws KryoException - IndexOutOfBoundsException

.write.buffer", 16097152) val table = new HTable(hConf, tableName) //table.setWriteBufferSize(8388608) *itr.grouped(100).foreach(table.put(_)) * // << Exception happens at this point table.close() } } I am using hbase 0.98.12 mapr distribution.

Re: Bulk loading Serialized RDD into Hbase throws KryoException - IndexOutOfBoundsException

bq. at com.mycorpt.myprojjobs.spark.jobs.hbase.HbaseUtils$$ anonfun$writeRddToHBase2$1.apply(HbaseUtils.scala:80) Can you reveal related code from HbaseUtils.scala ? Which hbase version are you using ? Thanks On Sun, May 29, 2016 at 4:26 PM, Nirav Patel wrote: > Hi, > > I a

Bulk loading Serialized RDD into Hbase throws KryoException - IndexOutOfBoundsException

Hi, I am getting following Kryo deserialization error when trying to buklload Cached RDD into Hbase. It works if I don't cache the RDD. I cache it with MEMORY_ONLY_SER. here's the code snippet: hbaseRdd.values.foreachPartition{ itr => val hConf = HBaseConfigu

Re: HBase / Spark Kerberos problem

Some of the Hadoop services cannot make use of the ticket obtained by oginUserFromKeytab. I was able to get past it using gss Jaas configuration where you can pass either Keytab file or ticketCache to spark executors that access HBase. Sent from my iPhone > On May 19, 2016, at 4:51 AM, El

RE: HBase / Spark Kerberos problem

Thanks Tom & John! modifying spark-env.sh did the trick - my last line in the file is now: export SPARK_DIST_CLASSPATH=$(paste -sd: "$SELF/classpath.txt"):`hbase classpath`:/etc/hbase/conf:/etc/hbase/conf/hbase-site.xml Now o.a.s.d.y.Client logs “Added HBase security token to cr

RE: HBase / Spark Kerberos problem

Yeah we ran into this issue. Key part is to have the hbase jars and hbase-site.xml config on the classpath of the spark submitter. We did it slightly differently from Y Bodnar, where we set the required jars and config on the env var SPARK_DIST_CLASSPATH in our spark env file (rather than

Re: HBase / Spark Kerberos problem

Have you had a look at this issue? https://issues.apache.org/jira/browse/SPARK-12279 There is a comment by Y Bodnar on how they successfully got Kerberos and HBase working. 2016-05-18 18:13 GMT+10:00 : > Hi all, > > I have been puzzling over a Kerberos problem for a while now and wond

HBase / Spark Kerberos problem

Hi all, I have been puzzling over a Kerberos problem for a while now and wondered if anyone can help. For spark-submit, I specify --master yarn-client --keytab x --principal y, which creates my SparkContext fine. Connections to Zookeeper Quorum to find the HBase master work well too. But when

HBase / Spark Kerberos problem

Hi all, I have been puzzling over a Kerberos problem for a while now and wondered if anyone can help. For spark-submit, I specify --keytab x --principal y, which creates my SparkContext fine. Connections to Zookeeper Quorum to find the HBase master work well too. But when it comes to a .count

Re: Save DataFrame to HBase

Please note: The name of hbase table is specified in: def writeCatalog = s"""{ |"table":{"namespace":"default", "name":"table1"}, not by the: HBaseTableCatalog.newTable -> "5" FYI On Tue,

Re: Save DataFrame to HBase

I think so. Please refer to the table population tests in (master branch): hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/DefaultSourceSuite.scala Cheers On Tue, May 10, 2016 at 2:53 PM, Benjamin Kim wrote: > Ted, > > Will the hbase-spark module allow for creating tables

Re: Save DataFrame to HBase

Ted, Will the hbase-spark module allow for creating tables in Spark SQL that reference the hbase tables underneath? In this way, users can query using just SQL. Thanks, Ben > On Apr 28, 2016, at 3:09 AM, Ted Yu wrote: > > Hbase 2.0 release likely would come after Spark 2.

Re: Save DataFrame to HBase

Hbase 2.0 release likely would come after Spark 2.0 release. There're other features being developed in hbase 2.0 I am not sure when hbase 2.0 would be released. The refguide is incomplete. Zhan has assigned the doc JIRA to himself. The documentation would be done after fixing bugs in

Re: Save DataFrame to HBase

Hi Ted, Do you know when the release will be? I also see some documentation for usage of the hbase-spark module at the hbase website. But, I don’t see an example on how to save data. There is only one for reading/querying data. Will this be added when the final version does get released

Re: Save DataFrame to HBase

t;> Hi Daniel, >> >> How did you get the Phoenix plugin to work? I have CDH 5.5.2 installed which >> comes with HBase 1.0.0 and Phoenix 4.5.2. Do you think this will work? >> >> Thanks, >> Ben >> >>> On Apr 24, 2016, at 1:43 AM, Daniel H

Re: Save DataFrame to HBase

be able to get the code > I've used for that project. > > Thank you. > Daniel > > On 24 Apr 2016, at 17:35, Benjamin Kim wrote: > > Hi Daniel, > > How did you get the Phoenix plugin to work? I have CDH 5.5.2 installed > which comes with HBase 1.0.0 and Phoen

Re: Save DataFrame to HBase

CDH 5.5.2 installed which > comes with HBase 1.0.0 and Phoenix 4.5.2. Do you think this will work? > > Thanks, > Ben > >> On Apr 24, 2016, at 1:43 AM, Daniel Haviv >> wrote: >> >> Hi, >> I tried saving DF to HBase using a hive table with hba

Re: Save DataFrame to HBase

Hi Daniel, How did you get the Phoenix plugin to work? I have CDH 5.5.2 installed which comes with HBase 1.0.0 and Phoenix 4.5.2. Do you think this will work? Thanks, Ben > On Apr 24, 2016, at 1:43 AM, Daniel Haviv > wrote: > > Hi, > I tried saving DF to HBase using a hive t

Re: Save DataFrame to HBase

Hi, I tried saving DF to HBase using a hive table with hbase storage handler and hiveContext but it failed due to a bug. I was able to persist the DF to hbase using Apache Pheonix which was pretty simple. Thank you. Daniel > On 21 Apr 2016, at 16:52, Benjamin Kim wrote: > > Has any

Re: Using saveAsNewAPIHadoopDataset for Saving custom classes to Hbase

Which hbase release are you using ? Below is the write method from hbase 1.1 : public void write(KEY key, Mutation value) throws IOException { if (!(value instanceof Put) && !(value instanceof Delete)) { throw new IOException("Pass a De

Using saveAsNewAPIHadoopDataset for Saving custom classes to Hbase

Hi All, I ma having a few issues saving my data to Hbase. I have created a pairRDD for my custom class using the following: val rdd1 =rdd.map{it=> (getRowKey(it), it) } val job = Job.getInstance(hConf) val jobConf = job.getConfiguration jobConf.

Re: Save DataFrame to HBase

You can try this https://github.com/hortonworks/shc.git or here http://spark-packages.org/package/zhzhan/shc Currently it is in the process of merging into HBase. Thanks. Zhan Zhang On Apr 21, 2016, at 8:44 AM, Benjamin Kim mailto:bbuil...@gmail.com>> wrote: Hi Ted, Can this mod

Re: Save DataFrame to HBase

Hi Ted, Can this module be used with an older version of HBase, such as 1.0 or 1.1? Where can I get the module from? Thanks, Ben > On Apr 21, 2016, at 6:56 AM, Ted Yu wrote: > > The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can do > this. > > On

Re: Save DataFrame to HBase

The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can do this. On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim wrote: > Has anyone found an easy way to save a DataFrame into HBase? > > Than

Save DataFrame to HBase

Has anyone found an easy way to save a DataFrame into HBase? Thanks, Ben - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

HBase Spark Module

I see that the new CDH 5.7 has been release with the HBase Spark module built-in. I was wondering if I could just download it and use the hbase-spark jar file for CDH 5.5. Has anyone tried this yet? Thanks, Ben - To unsubscribe

Re: ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper.

Please send query to user@hbase This is the default value: zookeeper.znode.parent /hbase Looks like hbase-site.xml accessible on your client didn't have up-to-date value for zookeeper.znode.parent Please make sure hbase-site.xml with proper config is on the classpath. On Sat, A

Re: [ERROR]: Spark 1.5.2 + Hbase 1.1 + Hive 1.2 + HbaseIntegration

forward you this mails, hope these can help you, you can take a look at this post http://www.abcn.net/2014/07/lighting-spark-with-hbase-full-edition.html 2016-03-04 3:30 GMT+01:00 Divya Gehlot : > Hi Teng, > > Thanks for the link you shared , helped me figure out the missing > depen

Re: build spark 1.6 against cdh5.7 with hadoop 2.6.0 hbase 1.2: Failure

-against-cdh5-7-with-hadoop-2-6-0-hbase-1-2-Failure-tp26762p26763.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e

build spark 1.6 against cdh5.7 with hadoop 2.6.0 hbase 1.2: Failure

ddress is not a member of object java.net.InetAddress [error] val s = new Socket(InetAddress.getLoopbackAddress(), port.get) [error] Thanks in advance. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/build-spark-1-6-against-cdh5-7-with-hadoop-2-6-0-hbase-1-2-Fa

Re: [ERROR]: Spark 1.5.2 + Hbase 1.1 + Hive 1.2 + HbaseIntegration

rg.apache.spark.sql._ > > val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) > hiveContext.sql("from TEST SELECT NAME").collect.foreach(println) > > > *Starting Spark shell* > spark-shell --jars > /usr/hdp/2.3.4.0-3485/hive/lib/guava-14.0.1.jar,/usr/hdp/2.3.4.0-348

Re: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray error in nelwy build Hbase

Dropping dev@ Can you provide a bit more information ? release of hbase release of hadoop I assume you're running on Linux. Any change in Linux setup before the exception showed up ? On Mon, Mar 28, 2016 at 10:30 AM, beeshma r wrote: > Hi > i am testing with newly build Hbase

A problem involving Spark & HBase.

Disclaimer: This is more of a design question. I am very new to Spark and HBase. This is going to be my first project using these 2 technologies and so far in last 2 months or so I’ve been just going over different resources to have a grasp on Spark and HBase. My question concerns mainly in terms

Re: Spark Job on YARN accessing Hbase Table

The backport would be done under HBASE-14160. FYI On Sun, Mar 13, 2016 at 4:14 PM, Benjamin Kim wrote: > Ted, > > Is there anything in the works or are there tasks already to do the > back-porting? > > Just curious. > > Thanks, > Ben > > On Mar 13, 2016, at

Re: Spark Job on YARN accessing Hbase Table

gt; Ted, > > I did as you said, but it looks like that HBaseContext relies on some > differences in HBase itself. > > [ERROR] > /home/bkim/hbase-rel-1.0.2/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala:30: > error: object HFileWriterImpl is no

Re: Spark Job on YARN accessing Hbase Table

, 2016 at 1:35 PM, Benjamin Kim wrote: > Ted, > > I did as you said, but it looks like that HBaseContext relies on some > differences in HBase itself. > > [ERROR] > /home/bkim/hbase-rel-1.0.2/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scal

Re: Spark Job on YARN accessing Hbase Table

Ted, I did as you said, but it looks like that HBaseContext relies on some differences in HBase itself. [ERROR] /home/bkim/hbase-rel-1.0.2/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala:30: error: object HFileWriterImpl is not a member of package

Re: Spark Job on YARN accessing Hbase Table

Ted, That’s great! I didn’t know. I will proceed with it as you said. Thanks, Ben > On Mar 13, 2016, at 12:42 PM, Ted Yu wrote: > > Benjamin: > Since hbase-spark is in its own module, you can pull the whole hbase-spark > subtree into hbase 1.0 root dir and add the following

Re: Spark Job on YARN accessing Hbase Table

Benjamin: Since hbase-spark is in its own module, you can pull the whole hbase-spark subtree into hbase 1.0 root dir and add the following to root pom.xml: hbase-spark Then you would be able to build the module yourself. hbase-spark module uses APIs which are compatible with hbase 1.0

Re: Spark Job on YARN accessing Hbase Table

Hi Ted, I see that you’re working on the hbase-spark module for hbase. I recently packaged the SparkOnHBase project and gave it a test run. It works like a charm on CDH 5.4 and 5.5. All I had to do was add /opt/cloudera/parcels/CDH/jars/htrace-core-3.1.0-incubating.jar to the classpath.txt

Re: How to obtain JavaHBaseContext to connection SparkStreaming with HBase

bq. Question is how to get maven repository As you may have noted, version has SNAPSHOT in it. Please checkout latest code from master branch and build it yourself. 2.0 release is still a few months away - though backport of hbase-spark module should come in 1.3 release. On Wed, Mar 9, 2016 at

How to obtain JavaHBaseContext to connection SparkStreaming with HBase

I am trying to integrate SparkStreaming with HBase. I am calling following APIs to connect to HBase HConnection hbaseConnection = HConnectionManager.createConnection(conf); hBaseTable = hbaseConnection.getTable(hbaseTable); Since I cannot get the connection and broadcast the connection each

Re: HBASE

om On 9 March 2016 at 15:45, Ted Yu wrote: > bq. it is kind of columnar NoSQL database. > > The storage format in HBase is not columnar. > > I would suggest you build upon what you already know (Spark and Hive) and > expand on that. Also, if your work uses Big Data technolog

Re: HBASE

bq. it is kind of columnar NoSQL database. The storage format in HBase is not columnar. I would suggest you build upon what you already know (Spark and Hive) and expand on that. Also, if your work uses Big Data technologies, those would be the first to consider getting to know better. On Wed

HBASE

Hi Gurus, I am relatively new to Big Data and know some about Spark and Hive. I was wondering do I need to pick up skills on Hbase as well. I am not sure how it works but know that it is kind of columnar NoSQL database. I know it is good to know something new in Big Data space. Just wondering if

[Issue:]Getting null values for Numeric types while accessing hive tables (Registered on Hbase,created through Phoenix)

Hi, I am registering hive table on Hbase CREATE EXTERNAL TABLE IF NOT EXISTS TEST(NAME STRING,AGE INT) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,0:AGE") TBLPROPERTI

Re: [ERROR]: Spark 1.5.2 + Hbase 1.1 + Hive 1.2 + HbaseIntegration

and also make sure that hbase-site.xml is set in your classpath on all nodes, both master and workers, and also client. normally i put it into $SPARK_HOME/conf/ then the spark cluster will be started with this conf file. btw. @Ted, did you tried insert into hbase table with spark's HiveCo

Re: [ERROR]: Spark 1.5.2 + Hbase 1.1 + Hive 1.2 + HbaseIntegration

16/03/01 01:36:31 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, ip-xxx-xx-xx-xxx.ap-southeast-1.compute.internal): java.lang.RuntimeException: hbase-default.xml file seems to be for an older version of HBase (null), this version is 1.1.2.2.3.4.0-3485 The above was likely caused by some

Re: [ERROR]: Spark 1.5.2 + Hbase 1.1 + Hive 1.2 + HbaseIntegration

16/02/29 23:09:34 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=9 watcher=hconnection-0x26fa89a20x0, quorum=localhost:2181, baseZNode=/hbase Since baseZNode didn't match what you set in hbase-site.xml, the cause was likely that hbase-site.xml

Re: [ERROR]: Spark 1.5.2 + Hbase 1.1 + Hive 1.2 + HbaseIntegration

16/02/29 23:09:34 INFO ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) Is your cluster secure cluster ? bq. Trace : Was there any output after 'Trace :' ? Was hbase-site.xml accessible to your

[ERROR]: Spark 1.5.2 + Hbase 1.1 + Hive 1.2 + HbaseIntegration

Spark shell* spark-shell --jars /usr/hdp/2.3.4.0-3485/hive/lib/guava-14.0.1.jar,/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-protocol.jar,/usr/hdp/2

[Help]: Steps to access hive table + Spark 1.5.2 + HbaseIntegration + Hive 1.2 + Hbase 1.1

Hi, Can anybody help me by sharing the steps/examples How to connect to hive table(which is being created using HbaseIntegration ) through hivecontext in Spark I googled but couldnt find a single example/document . Would really a

Re: Hbase in spark

I know little about your use case. Did you mean that your data is relatively evenly distributed in Spark domain but showed skew in the bulk load phase ? On Fri, Feb 26, 2016 at 9:02 AM, Renu Yadav wrote: > Hi Ted, > > Thanks for the reply. I am using spark hbase module only but th

Re: Hbase in spark

In hbase, there is hbase-spark module which supports bulk load. This module is to be backported in the upcoming 1.3.0 release. There is some pending work, such as HBASE-15271 . FYI On Fri, Feb 26, 2016 at 8:50 AM, Renu Yadav wrote: > Has anybody implemented bulk load into hbase using sp

Hbase in spark

Has anybody implemented bulk load into hbase using spark? I need help to optimize its performance. Please help. Thanks & Regards, Renu Yadav

Re: Spark Job on YARN accessing Hbase Table

Yes Ted, spark.executor.extraClassPath will work if hbase client jars is present in all Spark Worker / NodeManager machines. spark.yarn.dist.files is the easier way, as hbase client jars can be copied from driver machine or hdfs into container / spark-executor classpath automatically. No need to

Re: Spark Job on YARN accessing Hbase Table

Have you tried adding hbase client jars to spark.executor.extraClassPath ? Cheers On Wed, Feb 10, 2016 at 12:17 AM, Prabhu Joseph wrote: > + Spark-Dev > > For a Spark job on YARN accessing hbase table, added all hbase client jars > into spark.yarn.dist.files, NodeManager wh

Re: Spark Job on YARN accessing Hbase Table

+ Spark-Dev For a Spark job on YARN accessing hbase table, added all hbase client jars into spark.yarn.dist.files, NodeManager when launching container i.e executor, does localization and brings all hbase-client jars into executor CWD, but still the executor tasks fail with ClassNotFoundException

Spark Job on YARN accessing Hbase Table

Hi All, When i do count on a Hbase table from Spark Shell which runs as yarn-client mode, the job fails at count(). MASTER=yarn-client ./spark-shell import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor, TableName} import org.apache.hadoop.hbase.client.HBaseAdmin import

Re: Error trying to get DF for Hive table stored HBase

Looks like this is related: HIVE-12406 FYI On Tue, Feb 2, 2016 at 1:40 PM, Doug Balog wrote: > I’m trying to create a DF for an external Hive table that is in HBase. > I get the a NoSuchMethodError > org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.initSerdeParams(Lorg/apache/ha

Error trying to get DF for Hive table stored HBase

I’m trying to create a DF for an external Hive table that is in HBase. I get the a NoSuchMethodError org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.initSerdeParams(Lorg/apache/hadoop/conf/Configuration;Ljava/util/Properties;Ljava/lang/String;)Lorg/apache/hadoop/hive/serde2/lazy

Re: Spark Standalone cluster job to connect Hbase is Stuck

>From your first email, it seems that you don't observer output from hbase client. Spark 1.2 was quite old, missing fixes for log4j such as SPARK-9826 Can you change the following line in HBase's conf/log4j.properties from: log4j.logger.org.apache.hadoop.h

Re: Spark Standalone cluster job to connect Hbase is Stuck

Thanks Ted for quick reply. I am using spark 1.2, exporting Hbase conf directory containing hbase-site.xml in HADOOP_CLASSPATH & SPARK_CLASSPATH. Do i need to do anything else? Issues in connecting to kerberos Hbase through spark yarn cluster is fixed spark 1.4+, so i am trying if it work

Re: Spark Standalone cluster job to connect Hbase is Stuck

Is the hbase-site.xml on the classpath of the worker nodes ? Which Spark release are you using ? Cheers On Mon, Feb 1, 2016 at 4:25 PM, sudhir patil wrote: > Spark job on Standalone cluster is Stuck, shows no logs after > "util.AkkaUtils: Connecting to HeartbeatReceiver" on

Spark Standalone cluster job to connect Hbase is Stuck

:) 1. Did anyone see similar issue? or any clues on what could be the reason? 2. How do you i increase debug or log level? to see whats actually happening? 3. Any clues or links on how to use kerborised Hbase in spark standalone?

Re: HBase 0.98.0 with Spark 1.5.3 issue in yarn-cluster mode

I tried --jars which supposedly does that but that did not work. On Fri, Jan 22, 2016 at 4:33 PM Ajinkya Kale wrote: > Hi Ted, > Is there a way for the executors to have the hbase-protocol jar on their > classpath ? > > On Fri, Jan 22, 2016 at 4:00 PM Ted Yu wrote: >

Re: HBase 0.98.0 with Spark 1.5.3 issue in yarn-cluster mode

Hi Ted, Is there a way for the executors to have the hbase-protocol jar on their classpath ? On Fri, Jan 22, 2016 at 4:00 PM Ted Yu wrote: > The class path formations on driver and executors are different. > > Cheers > > On Fri, Jan 22, 2016 at 3:25 PM, Ajinkya Kale > wrote:

Re: HBase 0.98.0 with Spark 1.5.3 issue in yarn-cluster mode

> I am not aware of a workaround. >>> >>> Can you upgrade to 0.98.4+ release ? >>> >>> Cheers >>> >>> On Wed, Jan 20, 2016 at 6:26 PM, Ajinkya Kale >>> wrote: >>> >>>> Hi Ted, >>>> >>>> Thank

Re: HBase 0.98.0 with Spark 1.5.3 issue in yarn-cluster mode

>> >>> Hi Ted, >>> >>> Thanks for responding. >>> Is there a work around for 0.98.0 ? Adding the hbase-protocol jar to >>> HADOOP_CLASSPATH didnt work for me. >>> >>> On Wed, Jan 20, 2016 at 6:14 PM Ted Yu wrote: >>> &

Re: HBase 0.98.0 with Spark 1.5.3 issue in yarn-cluster mode

>> Hi Ted, >> >> Thanks for responding. >> Is there a work around for 0.98.0 ? Adding the hbase-protocol jar to >> HADOOP_CLASSPATH didnt work for me. >> >> On Wed, Jan 20, 2016 at 6:14 PM Ted Yu wrote: >> >>> 0.98.0 didn't have fix from HB

Re: HBase 0.98.0 with Spark 1.5.3 issue in yarn-cluster mode

I am not aware of a workaround. Can you upgrade to 0.98.4+ release ? Cheers On Wed, Jan 20, 2016 at 6:26 PM, Ajinkya Kale wrote: > Hi Ted, > > Thanks for responding. > Is there a work around for 0.98.0 ? Adding the hbase-protocol jar to > HADOOP_CLASSPATH didnt work for me. >

Re: HBase 0.98.0 with Spark 1.5.3 issue in yarn-cluster mode

Hi Ted, Thanks for responding. Is there a work around for 0.98.0 ? Adding the hbase-protocol jar to HADOOP_CLASSPATH didnt work for me. On Wed, Jan 20, 2016 at 6:14 PM Ted Yu wrote: > 0.98.0 didn't have fix from HBASE-8 > > Please upgrade your hbase version and try again

Re: HBase 0.98.0 with Spark 1.5.3 issue in yarn-cluster mode

0.98.0 didn't have fix from HBASE-8 Please upgrade your hbase version and try again. If still there is problem, please pastebin the stack trace. Thanks On Wed, Jan 20, 2016 at 5:41 PM, Ajinkya Kale wrote: > > I have posted this on hbase user list but i thought makes more sense

HBase 0.98.0 with Spark 1.5.3 issue in yarn-cluster mode

I have posted this on hbase user list but i thought makes more sense on spark user list. I am able to read the table in yarn-client mode from spark-shell but I have exhausted all online forums for options to get it working in the yarn-cluster mode through spark-submit. I am using this code

Re: is Hbase Scan really need thorough Get (Hbase+solr+spark)

Thanks Ted, :) if everything gets indexed from Hbase into solr ,then no need to trace Regionservers once again Thanks Beesh On Wed, Jan 20, 2016 at 5:05 AM, Ted Yu wrote: > get(List gets) will call: > > Object [] r1 = batch((List)gets); > > where b

Re: is Hbase Scan really need thorough Get (Hbase+solr+spark)

. FYI On Tue, Jan 19, 2016 at 3:28 PM, ayan guha wrote: > It is not scanning the HBase. What it is doing is looping through your > list of Row keys and fetching data for each 1 at a time. > > Ex: Your solr result has 5 records, with Row Keys R1...R5. > Then list will be [R1,R2,

Re: is Hbase Scan really need thorough Get (Hbase+solr+spark)

It is not scanning the HBase. What it is doing is looping through your list of Row keys and fetching data for each 1 at a time. Ex: Your solr result has 5 records, with Row Keys R1...R5. Then list will be [R1,R2,...R5] Then table.get(list) will do something like: res=[] for k in list: v

is Hbase Scan really need thorough Get (Hbase+solr+spark)

Hi I trying to integrated Hbase-solr-spark. Solr is indexing all the documents from Hbase through hbase-indexer . Through the Spark I am manipulating all datasets .Thing is after getting the solrdocuments from the solr query ,it has the rowkey and rowvalues .So directly i got the rowkeys and

Re: Error in Spark Executors when trying to read HBase table from Spark with Kerberos enabled

Hi Guys, Any help regarding this issue..?? On Wed, Jan 13, 2016 at 6:39 PM, Vinay Kashyap wrote: > Hi all, > > I am using *Spark 1.5.1 in YARN cluster mode in CDH 5.5.* > I am trying to create an RDD by reading HBase table with kerberos enabled. > I am able to launch the sp

Re: Spark and HBase RDD join/get

Thanks Ted! On Thu, Jan 14, 2016 at 4:49 PM, Ted Yu wrote: > For #1, yes it is possible. > > You can find some example in hbase-spark module of hbase where hbase as > DataSource is provided. > e.g. > > https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/or

Re: Spark and HBase RDD join/get

For #1, yes it is possible. You can find some example in hbase-spark module of hbase where hbase as DataSource is provided. e.g. https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseRDDFunctions.scala Cheers On Thu, Jan 14, 2016 at 5:04 AM

Spark and HBase RDD join/get

Hi We have a RDD that needs to be mapped with information from HBase, where the exact key is the user id. What's the different alternatives for doing this? - Is it possible to do HBase.get() requests from a map function in Spark? - Or should we join RDDs with all full HBase table scan?

Error in Spark Executors when trying to read HBase table from Spark with Kerberos enabled

Hi all, I am using *Spark 1.5.1 in YARN cluster mode in CDH 5.5.* I am trying to create an RDD by reading HBase table with kerberos enabled. I am able to launch the spark job to read the HBase table, but I notice that the executors launched for the job cannot proceed due to an issue with

Re: How to insert df in HBASE

Cycling prior discussion: http://search-hadoop.com/m/q3RTtX7POh17hqdj1 On Wed, Jan 6, 2016 at 3:07 AM, Sadaf wrote: > HI, > > I need to insert a Dataframe in to hbase using scala code. > Can anyone guide me how to achieve this? > > Any help would be much app

How to insert df in HBASE

HI, I need to insert a Dataframe in to hbase using scala code. Can anyone guide me how to achieve this? Any help would be much appreciated. Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-insert-df-in-HBASE-tp25891.html Sent from the

Re: how to fetch all of data from hbase table in spark java

Please take a look at: examples/src/main/scala/org/apache/spark/examples/HBaseTest.scala There're various hbase connectors (search for 'apache spark hbase connector') In hbase 2.0, there would be hbase-spark module which provides hbase connector. FYI On Fri, Dec 18, 2015 at 11

<    1   2   3   4   5   6   7   8   >