Re: Hive metadata on Hbase

[Per Ullberg]

I thought the main gain was to get ACID on Hive performant enough.

@Mich: Do you run with ACID-enabled tables? How many Create/Update/Deletes
do you do per second?

best regards
/Pelle

On Mon, Oct 24, 2016 at 7:39 AM, Jörn Franke  wrote:

> I think the main gain is more about getting rid of a dedicated database
> including maintenance and potential license cost.
> For really large clusters and a lot of users this might be even more
> beneficial. You can avoid clustering the database etc.
>
> On 24 Oct 2016, at 00:46, Mich Talebzadeh 
> wrote:
>
>
> A while back there was some notes on having Hive metastore on Hbase as
> opposed to conventional RDBMSs
>
> I am currently involved with some hefty work with Hbase and Phoenix for
> batch ingestion of trade data. As long as you define your Hbase table
> through Phoenix and with secondary Phoenix indexes on Hbase, the speed is
> impressive.
>
> I am not sure how much having Hbase as Hive metastore is going to add to
> Hive performance. We use Oracle 12c as Hive metastore and the Hive
> database/schema is built on solid state disks. Never had any issues with
> lock and concurrency.
>
> Therefore I am not sure what one is going to gain by having Hbase as the
> Hive metastore? I trust that we can still use our existing schemas on
> Oracle.
>
> HTH
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>


-- 

*Per Ullberg*
Data Vault Tech Lead
Odin Uppsala
+46 701612693 <+46+701612693>

Klarna AB (publ)
Sveavägen 46, 111 34 Stockholm
Tel: +46 8 120 120 00 <+46812012000>
Reg no: 556737-0431
klarna.com

Re: Hive metadata on Hbase

[Jörn Franke]

I think the main gain is more about getting rid of a dedicated database 
including maintenance and potential license cost. 
For really large clusters and a lot of users this might be even more 
beneficial. You can avoid clustering the database etc.

> On 24 Oct 2016, at 00:46, Mich Talebzadeh  wrote:
> 
> 
> A while back there was some notes on having Hive metastore on Hbase as 
> opposed to conventional RDBMSs
> 
> I am currently involved with some hefty work with Hbase and Phoenix for batch 
> ingestion of trade data. As long as you define your Hbase table through 
> Phoenix and with secondary Phoenix indexes on Hbase, the speed is impressive.
> 
> I am not sure how much having Hbase as Hive metastore is going to add to Hive 
> performance. We use Oracle 12c as Hive metastore and the Hive database/schema 
> is built on solid state disks. Never had any issues with lock and concurrency.
> 
> Therefore I am not sure what one is going to gain by having Hbase as the Hive 
> metastore? I trust that we can still use our existing schemas on Oracle.
> 
> HTH
> 
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
> damage or destruction of data or any other property which may arise from 
> relying on this email's technical content is explicitly disclaimed. The 
> author will in no case be liable for any monetary damages arising from such 
> loss, damage or destruction.
>  

Hive metadata on Hbase

[Mich Talebzadeh]

A while back there was some notes on having Hive metastore on Hbase as
opposed to conventional RDBMSs

I am currently involved with some hefty work with Hbase and Phoenix for
batch ingestion of trade data. As long as you define your Hbase table
through Phoenix and with secondary Phoenix indexes on Hbase, the speed is
impressive.

I am not sure how much having Hbase as Hive metastore is going to add to
Hive performance. We use Oracle 12c as Hive metastore and the Hive
database/schema is built on solid state disks. Never had any issues with
lock and concurrency.

Therefore I am not sure what one is going to gain by having Hbase as the
Hive metastore? I trust that we can still use our existing schemas on
Oracle.

HTH



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

Re: hiveserver2 GC overhead limit exceeded

[Stephen Sprague]

ok. i'll bite.

lets see the output of this command where Hiveserver2 is running.

$ ps -ef | grep -i hiveserver2

this'll show us all the command line parameters HS2 was (ultimately)
invoked with.

Cheers,
Stephen

On Sun, Oct 23, 2016 at 6:46 AM, patcharee 
wrote:

> Hi,
>
> I use beeline to connect to hiveserver2. I tested with a simple command
> and got an error GC overhead limit exceeded
>
> 0: jdbc:hive2://service-10-1:10010/default> drop table
> testhivedrivertable;
> Error: Error while processing statement: FAILED: Execution Error, return
> code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. GC overhead limit
> exceeded (state=08S01,code=1)
>
> How to solve this? How to identify if this error is from the client
> (beeline) or from hiveserver2?
>
> Thanks,
>
> Patcharee
>
>

hiveserver2 GC overhead limit exceeded

[patcharee]

Hi,

I use beeline to connect to hiveserver2. I tested with a simple command 
and got an error GC overhead limit exceeded


0: jdbc:hive2://service-10-1:10010/default> drop table testhivedrivertable;
Error: Error while processing statement: FAILED: Execution Error, return 
code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. GC overhead limit 
exceeded (state=08S01,code=1)


How to solve this? How to identify if this error is from the client 
(beeline) or from hiveserver2?


Thanks,

Patcharee