Re: There is any way to write from spark to HBase CDH4?
You can get a feel for it by playing with the original library published as
separate project on github
https://github.com/cloudera-labs/SparkOnHBase
From: Deng Ching-Mallete
Date: Tuesday, October 27, 2015 at 12:39 PM
To: Fengdong Yu
Cc: Adrian Tanase, avivb, user
Subject: Re: There is any way to write from spark to HBase CDH4?
It's still in HBase' trunk, scheduled for 2.0.0 release based on Jira ticket.
-Deng
On Tue, Oct 27, 2015 at 6:35 PM, Fengdong Yu
mailto:fengdo...@everstring.com>> wrote:
Does this released with Spark1.*? or still kept in the trunk?
On Oct 27, 2015, at 6:22 PM, Adrian Tanase
mailto:atan...@adobe.com>> wrote:
Also I just remembered about cloudera’s contribution
http://blog.cloudera.com/blog/2015/08/apache-spark-comes-to-apache-hbase-with-hbase-spark-module/
From: Deng Ching-Mallete
Date: Tuesday, October 27, 2015 at 12:03 PM
To: avivb
Cc: user
Subject: Re: There is any way to write from spark to HBase CDH4?
Hi,
We are using phoenix-spark (http://phoenix.apache.org/phoenix_spark.html) to
write data to HBase, but it requires spark 1.3.1+ and phoenix 4.4+. Previously,
when we were still on spark 1.2, we used the HBase API to write directly to
HBase.
For HBase 0.98, it's something like this:
rdd.foreachPartition(partition => {
// create hbase config
val hConf = HBaseConfiguration.create()
val hTable = new HTable(hConf, "TABLE_1")
hTable.setAutoFlush(false)
partition.foreach(r => {
// generate row key
// create row
val hRow = new Put(rowKey)
// add columns
hRow.add(..)
hTable.put(hRow)
})
hTable.flushCommits()
hTable.close()
})
HTH,
Deng
On Tue, Oct 27, 2015 at 5:36 PM, avivb
mailto:a...@taykey.com>> wrote:
I have already try it with https://github.com/unicredit/hbase-rdd and
https://github.com/nerdammer/spark-hbase-connector and in both cases I get
timeout.
So I would like to know about other option to write from Spark to HBase
CDH4.
Thanks!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/There-is-any-way-to-write-from-spark-to-HBase-CDH4-tp25209.html
Sent from the Apache Spark User List mailing list archive at
Nabble.com<http://Nabble.com>.
-
To unsubscribe, e-mail:
user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org>
For additional commands, e-mail: user-h...@spark.apache.org
<mailto:user-h...@spark.apache.org>
Re: There is any way to write from spark to HBase CDH4?
It's still in HBase' trunk, scheduled for 2.0.0 release based on Jira
ticket.
-Deng
On Tue, Oct 27, 2015 at 6:35 PM, Fengdong Yu
wrote:
> Does this released with Spark1.*? or still kept in the trunk?
>
>
>
>
> On Oct 27, 2015, at 6:22 PM, Adrian Tanase wrote:
>
> Also I just remembered about cloudera’s contribution
>
> http://blog.cloudera.com/blog/2015/08/apache-spark-comes-to-apache-hbase-with-hbase-spark-module/
>
> From: Deng Ching-Mallete
> Date: Tuesday, October 27, 2015 at 12:03 PM
> To: avivb
> Cc: user
> Subject: Re: There is any way to write from spark to HBase CDH4?
>
> Hi,
>
> We are using phoenix-spark (http://phoenix.apache.org/phoenix_spark.html)
> to write data to HBase, but it requires spark 1.3.1+ and phoenix 4.4+.
> Previously, when we were still on spark 1.2, we used the HBase API to write
> directly to HBase.
>
> For HBase 0.98, it's something like this:
>
> rdd.foreachPartition(partition => {
>// create hbase config
>val hConf = HBaseConfiguration.create()
>val hTable = new HTable(hConf, "TABLE_1")
>hTable.setAutoFlush(false)
>
>partition.foreach(r => {
> // generate row key
> // create row
> val hRow = new Put(rowKey)
>
> // add columns
> hRow.add(..)
>
> hTable.put(hRow)
>})
>hTable.flushCommits()
>hTable.close()
> })
>
> HTH,
> Deng
>
> On Tue, Oct 27, 2015 at 5:36 PM, avivb wrote:
>
>> I have already try it with https://github.com/unicredit/hbase-rdd and
>> https://github.com/nerdammer/spark-hbase-connector and in both cases I
>> get
>> timeout.
>>
>> So I would like to know about other option to write from Spark to HBase
>> CDH4.
>>
>> Thanks!
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/There-is-any-way-to-write-from-spark-to-HBase-CDH4-tp25209.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>
Re: There is any way to write from spark to HBase CDH4?
Does this released with Spark1.*? or still kept in the trunk?
> On Oct 27, 2015, at 6:22 PM, Adrian Tanase wrote:
>
> Also I just remembered about cloudera’s contribution
> http://blog.cloudera.com/blog/2015/08/apache-spark-comes-to-apache-hbase-with-hbase-spark-module/
>
> <http://blog.cloudera.com/blog/2015/08/apache-spark-comes-to-apache-hbase-with-hbase-spark-module/>
>
> From: Deng Ching-Mallete
> Date: Tuesday, October 27, 2015 at 12:03 PM
> To: avivb
> Cc: user
> Subject: Re: There is any way to write from spark to HBase CDH4?
>
> Hi,
>
> We are using phoenix-spark (http://phoenix.apache.org/phoenix_spark.html
> <http://phoenix.apache.org/phoenix_spark.html>) to write data to HBase, but
> it requires spark 1.3.1+ and phoenix 4.4+. Previously, when we were still on
> spark 1.2, we used the HBase API to write directly to HBase.
>
> For HBase 0.98, it's something like this:
>
> rdd.foreachPartition(partition => {
>// create hbase config
>val hConf = HBaseConfiguration.create()
>val hTable = new HTable(hConf, "TABLE_1")
>hTable.setAutoFlush(false)
>
>partition.foreach(r => {
> // generate row key
> // create row
> val hRow = new Put(rowKey)
>
> // add columns
> hRow.add(..)
>
> hTable.put(hRow)
>})
>hTable.flushCommits()
>hTable.close()
> })
>
> HTH,
> Deng
>
> On Tue, Oct 27, 2015 at 5:36 PM, avivb <mailto:a...@taykey.com>> wrote:
>> I have already try it with https://github.com/unicredit/hbase-rdd
>> <https://github.com/unicredit/hbase-rdd> and
>> https://github.com/nerdammer/spark-hbase-connector
>> <https://github.com/nerdammer/spark-hbase-connector> and in both cases I get
>> timeout.
>>
>> So I would like to know about other option to write from Spark to HBase
>> CDH4.
>>
>> Thanks!
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/There-is-any-way-to-write-from-spark-to-HBase-CDH4-tp25209.html
>>
>> <http://apache-spark-user-list.1001560.n3.nabble.com/There-is-any-way-to-write-from-spark-to-HBase-CDH4-tp25209.html>
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> <mailto:user-unsubscr...@spark.apache.org>
>> For additional commands, e-mail: user-h...@spark.apache.org
>> <mailto:user-h...@spark.apache.org>
Re: There is any way to write from spark to HBase CDH4?
Also I just remembered about cloudera’s contribution
http://blog.cloudera.com/blog/2015/08/apache-spark-comes-to-apache-hbase-with-hbase-spark-module/
From: Deng Ching-Mallete
Date: Tuesday, October 27, 2015 at 12:03 PM
To: avivb
Cc: user
Subject: Re: There is any way to write from spark to HBase CDH4?
Hi,
We are using phoenix-spark (http://phoenix.apache.org/phoenix_spark.html) to
write data to HBase, but it requires spark 1.3.1+ and phoenix 4.4+. Previously,
when we were still on spark 1.2, we used the HBase API to write directly to
HBase.
For HBase 0.98, it's something like this:
rdd.foreachPartition(partition => {
// create hbase config
val hConf = HBaseConfiguration.create()
val hTable = new HTable(hConf, "TABLE_1")
hTable.setAutoFlush(false)
partition.foreach(r => {
// generate row key
// create row
val hRow = new Put(rowKey)
// add columns
hRow.add(..)
hTable.put(hRow)
})
hTable.flushCommits()
hTable.close()
})
HTH,
Deng
On Tue, Oct 27, 2015 at 5:36 PM, avivb
mailto:a...@taykey.com>> wrote:
I have already try it with https://github.com/unicredit/hbase-rdd and
https://github.com/nerdammer/spark-hbase-connector and in both cases I get
timeout.
So I would like to know about other option to write from Spark to HBase
CDH4.
Thanks!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/There-is-any-way-to-write-from-spark-to-HBase-CDH4-tp25209.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail:
user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org>
For additional commands, e-mail: user-h...@spark.apache.org
<mailto:user-h...@spark.apache.org>
Re: There is any way to write from spark to HBase CDH4?
Hi,
We are using phoenix-spark (http://phoenix.apache.org/phoenix_spark.html)
to write data to HBase, but it requires spark 1.3.1+ and phoenix 4.4+.
Previously, when we were still on spark 1.2, we used the HBase API to write
directly to HBase.
For HBase 0.98, it's something like this:
rdd.foreachPartition(partition => {
// create hbase config
val hConf = HBaseConfiguration.create()
val hTable = new HTable(hConf, "TABLE_1")
hTable.setAutoFlush(false)
partition.foreach(r => {
// generate row key
// create row
val hRow = new Put(rowKey)
// add columns
hRow.add(..)
hTable.put(hRow)
})
hTable.flushCommits()
hTable.close()
})
HTH,
Deng
On Tue, Oct 27, 2015 at 5:36 PM, avivb wrote:
> I have already try it with https://github.com/unicredit/hbase-rdd and
> https://github.com/nerdammer/spark-hbase-connector and in both cases I get
> timeout.
>
> So I would like to know about other option to write from Spark to HBase
> CDH4.
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/There-is-any-way-to-write-from-spark-to-HBase-CDH4-tp25209.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
Re: There is any way to write from spark to HBase CDH4?
This is probably too low level but you could consider the async client inside foreachRdd: https://github.com/OpenTSDB/asynchbase http://spark.apache.org/docs/latest/streaming-programming-guide.html#design-patterns-for-using-foreachrdd On 10/27/15, 11:36 AM, "avivb" wrote: >I have already try it with https://github.com/unicredit/hbase-rdd and >https://github.com/nerdammer/spark-hbase-connector and in both cases I get >timeout. > >So I would like to know about other option to write from Spark to HBase >CDH4. > >Thanks! > > > >-- >View this message in context: >http://apache-spark-user-list.1001560.n3.nabble.com/There-is-any-way-to-write-from-spark-to-HBase-CDH4-tp25209.html >Sent from the Apache Spark User List mailing list archive at Nabble.com. > >- >To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >For additional commands, e-mail: user-h...@spark.apache.org >
There is any way to write from spark to HBase CDH4?
I have already try it with https://github.com/unicredit/hbase-rdd and https://github.com/nerdammer/spark-hbase-connector and in both cases I get timeout. So I would like to know about other option to write from Spark to HBase CDH4. Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/There-is-any-way-to-write-from-spark-to-HBase-CDH4-tp25209.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
