Re: Hive Metastore Hook to to fire only on success

ly see the > commitCreateTable hook call if the creation succeeds. > > Alan. > > On Thu, Oct 4, 2018 at 12:36 AM Daniel Haviv > wrote: > >> Hi, >> I'm writing a HMS hook and I noticed that the hook fires no matter if the >> operation succeeded or not. >> For e

Hive Metastore Hook to to fire only on success

Hi, I'm writing a HMS hook and I noticed that the hook fires no matter if the operation succeeded or not. For example, if a user creates an already existing table, the operation will fail but the the hook will fire regardless. Is there a way to either validate that the operation succeeded or fire

Specifying orc.stripe.size in Spark

Hi, When writing a dataframe using: df.write.orc("/path/to/orc") How can I specify orc parameters like orc.stripe.size ? Thank you, Daniel

Column names in ORC file

Hi, When I'm generating ORC files using spark the column names are written into the ORC file but when generated using Hive I get the following column names: _col107, _col33, _col23, _col102 Is it possible to somehow configure hive to properly store the column names like Spark? Thank you,

Re: IntWritable cannot be cast to LongWritable

.1 should automatically handle the > type conversions from the file to the table. > > .. Owen > > On Wed, Dec 14, 2016 at 9:36 AM, Daniel Haviv < > daniel.ha...@veracity-group.com> wrote: > >> Hi, >> I have an ORC table where one of the fields was an int and is now a &

IntWritable cannot be cast to LongWritable

Hi, I have an ORC table where one of the fields was an int and is now a bigint. Whenever I query a partition before the schema change I encounter the following error: Error: java.io.IOException: java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to

Re: How to setup Hive JDBC client to connect remote Hiveserver

you please suggest me the > exact location. > > Thanks > Braj > >> On Mon, Apr 4, 2016 at 10:25 AM, Daniel Haviv >> <daniel.ha...@veracity-group.com> wrote: >> It seems your hive server is not up (or not listening on port 1). >> hiveserver's logs m

Re: How to setup Hive JDBC client to connect remote Hiveserver

It seems your hive server is not up (or not listening on port 1). hiveserver's logs might shed some light (usually at /var/log/hive) Thank you. Daniel > On 4 Apr 2016, at 07:00, brajmohan saxena wrote: > > Hi Shumin, > > I did telnet > > braj-laptop:bin

Re: Hive_CSV

Hi Ajay, Use the CSV serde to read your file, map all three columns but only select the relevant ones when you insert: Create table csvtab ( irrelevant string, sportName string, sportType string) ... Insert into loaded_table select sportName, sportType from csvtab; Daniel > On 9 Mar 2016, at

Partition level inputformat

Hi, I'm trying to add external partitions to a table with a different inputformat and row delimiter properties but I keep failing and I can't find any documentation that explains the correct syntax. This is the DML I'm running: hive> alter table test_tbl_parquet add partition

Re: chmod: changing permissions of '/datadir/000056_0': Permission denied. user=danielh is not the owner of inode=000056_0

Hi, Any thoughts on this issue ? Thank you. Daniel On Wed, Jan 20, 2016 at 12:28 PM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > Hi, > We have a table in which the files are created by different users (under > the same group). > When a user inserts into the ta

chmod: changing permissions of '/datadir/000056_0': Permission denied. user=danielh is not the owner of inode=000056_0

Hi, We have a table in which the files are created by different users (under the same group). When a user inserts into the table it will finish successfully but after moving the files the user will receive the following error(s): chmod: changing permissions of '/datadir/56_0': Permission

Fwd: Conversion

Hi, We have a string column that represents an array of doubles that looks like this: f7 ad 3b 38 89 b7 e5 3f a1 c1 1a 74 db To parse it we use unhex(translate(signalvalues,' ','')) which returns a BINARY value. How can we convert it to ARRAY ? Thank you. Daniel

simple usage of stack UDTF causes a cast exception

Hi, I'm trying to break a row into two rows based on two different columns by using the following query: SELECT mystack.alias1 FROM cdrtable LATERAL VIEW stack(2, caller_IMEI, recipient_IMEI) mystack AS alias1; The exception I'm hitting is: java.io.IOException:

Re: UPDATE RE: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes (beeline - hive server 2)

Hi, I remember encountering a similar problem that was caused by an old mysql client driver. You can try and upgrade your mysql connector. Daniel On Mon, Nov 30, 2015 at 8:12 PM, Timothy Garza < timothy.ga...@collinsongroup.com> wrote: > We’ve been playing with the MySQL Global Settings: (Hive

Re: Hive On Spark - Using custom SerDe

uot;? You also need > to make sure your jar is in ""hive.aux.jars.path". > > Thanks, > Jimmy > > On Mon, Nov 16, 2015 at 1:44 AM, Daniel Haviv < > daniel.ha...@veracity-group.com> wrote: > >> Hi, >> We have a custom SerDe we would like to use with Hive

Hive On Spark - Using custom SerDe

Hi, We have a custom SerDe we would like to use with Hive on Spark but I'm not sure how to. The error messages are pretty clear about the fact that it can't find my SerDE's class: Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class:

Re: Disabling local mode optimization

queries with multiple > map-reduce jobs where the input to subsequent jobs is substantially smaller > (because of reduction/filtering in the prior job), jobs may be run locally. > > so we may need to check the sizeof your input, which version of hive are > you using? it can work only

Re: Merging small files

Thanks for the tip Gopal. I tried what you suggested (on Tez) but I'm getting a middle stage with 1 reducer (which is awful for performance). This is my query: insert into upstreamparam_org partition(day_ts, cmtsid) select * from upstreamparam_20151013 order by datats,macaddress; I've attached

Re: Merging small files

Changed it to sort by. On Sat, Oct 17, 2015 at 6:05 PM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > Thanks for the tip Gopal. > I tried what you suggested (on Tez) but I'm getting a middle stage with 1 > reducer (which is awful for performance). > > This

Merging small files

Hi, We are using Hive to merge small files by setting hive.merge.smallfiles.avgsize to 12000 and doing an insert as select to a table. The problem is that this take two passes over the data, first to insert the data and then to merge it. Is there a more efficient way to have Hive merge small

Re: Hive SerDe regex error

Hi, You didn't escape the ^ character at the end. Try using this string instead: ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ \[]*)\[([^ ]*)\]: \(([^ ]*)\) ([\^]*) Daniel On Thu, Oct 1, 2015 at 3:17 PM, IT CTO wrote: > Hi, > I am trying to create a table with Regex SerDe but failing

Re: Error: java.lang.IllegalArgumentE:Column has wrong number of index entries found - when trying to insert from JSON external table to ORC table

Hi Prasanth, Can you elaborate on what does the hive.merge.orcfile.stripe.level parameter affext? Thank you for your help. Daniel Sent from my iPhone > On 8 בספט׳ 2015, at 17:48, Prasanth Jayachandran > wrote: > > hive.merge.orcfile.stripe.level

Permission denied error when starting HiveServer2

Hi, I'm getting this error when starting HiveServer2: 2015-09-07 08:09:50,356 WARN org.apache.hive.service.server.HiveServer2: Error starting HiveServer2 on attempt 1, will retry in 60 seconds java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Permission denied at

Re: Disabling local mode optimization

> so we may need to check the sizeof your input, which version of hive are you > using? it can work only from Hive 0.7 onwards > >> On Wed, Sep 2, 2015 at 4:46 PM, Daniel Haviv >> <daniel.ha...@veracity-group.com> wrote: >> Hi, >> I would like to disable the o

Disabling local mode optimization

Hi, I would like to disable the optimization where a query that just selects data is running without mapreduce (local mode). hive.exec.mode.local.auto is set to false but hive still runs in local mode for some queries. How can I disable local mode completely? Thank you. Daniel

Re: Loading multiple file format in hive

Hi, You can set a different file format per partition. You can't mix files in the same directory (You could theoretically write some kind of custom SerDe). Daniel. On Mon, Aug 24, 2015 at 6:15 PM, Jeetendra G jeetendr...@housing.com wrote: Can anyone put some light on this please? On Mon,

Re: Output of Hive

It seems like your query returns no results,try using count to confirm. Daniel On 16 במאי 2015, at 14:40, Anand Murali anand_vi...@yahoo.com wrote: Dear All: I am new to hive so pardon my ignorance. I have the following query but do not see any output. I wondered it maybe in HDFS and

Re: Extremely Slow Data Loading with 40k+ Partitions

data to table default.parquet_table_with_40k_partitions partition (yearmonth=null, prefix=null) When I look at the size of hdfs files of table, I can see the size is growing, but it's kind of slow. For mapreduce job, I had 400+ mappers and 100+ reducers. Thanks Tianqi From: Daniel

Re: Extremely Slow Data Loading with 40k+ Partitions

How many reducers are you using? Daniel On 16 באפר׳ 2015, at 00:55, Tianqi Tong tt...@brightedge.com wrote: Hi, I'm loading data to a Parquet table with dynamic partitons. I have 40k+ partitions, and I have skipped the partition stats computation step. Somehow it's still exetremely slow

HiveServer2 addressing standby namenode

Hi, We get a lot of error messaged on the standby namenode indicating that hive is trying to address the standby namenode. As all of our jobs function normally, my guess is that Hive is constantly trying to address both namenodes and only works with the active one. Is this correct? Can this be

Re: hive 0.14 return some not NULL value as NULL

Can you also supply the table's DDL and a few lines of your raw data? Daniel On 1 באפר׳ 2015, at 09:16, r7raul1...@163.com r7raul1...@163.com wrote: I use hive 0.14 the result is 87FQEZT1UEDXJHJQPFFX7G7ET8S2DVPM2357378283356 9150119100048 7326356 NULL

Re: 0.14 parse exception, row format question

Your quotation marks around the location string seem to be wrong Daniel On 26 במרץ 2015, at 22:10, bitsofinfo bitsofinf...@gmail.com wrote: Hi, What is wrong with this query? I am reading the docs and it appears that this should work no? INSERT OVERWRITE DIRECTORY

Re: Understanding Hive's execution plan

subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Daniel Haviv [mailto:daniel.ha...@veracity-group.com

Re: how to set column level privileges

Create a view with the permitted columns and handle the privileges for it Daniel On 26 במרץ 2015, at 12:40, Allen bjallenw...@sina.com wrote: hi, We use SQL standards based authorization for authorization in Hive 0.14. But it has not support for column level privileges.

Understanding Hive's execution plan

Hi, Can anyone direct me to a good explanation on understanding Hive's execution plan? Thanks, Daniel

Re: How to clean up a table for which the underlying hdfs file no longer exists

You can (as a workaround) just create it's directory and then drop it Daniel On 22 במרץ 2015, at 04:15, Stephen Boesch java...@gmail.com wrote: There is a hive table for which the metadata points to a non-existing hdfs file. Simply calling drop table mytable results in:

Re: How to clean up a table for which the underlying hdfs file no longer exists

You can also use ALTER TABLE table SET TBLPROPERTIES('EXTERNAL'='TRUE') And then drop it Daniel On 22 במרץ 2015, at 04:15, Stephen Boesch java...@gmail.com wrote: There is a hive table for which the metadata points to a non-existing hdfs file. Simply calling drop table mytable

Re: insert table error

What is the error you get? Daniel On 13 במרץ 2015, at 13:13, zhangjp smart...@hotmail.com wrote: case fail CREATE TABLE students (name VARCHAR(64), age INT, gpa DECIMAL(3, 2)) CLUSTERED BY (age) INTO 2 BUCKETS STORED AS ORC; INSERT INTO TABLE students VALUES ('fred flintstone', 35,

Re: Which SerDe for Custom Binary Data.

https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide#DeveloperGuide-HowtoWriteYourOwnSerDe Daniel On 13 במרץ 2015, at 17:56, karthik maddala karthikmaddal...@gmail.com wrote: I want to set up a DW based on Hive. However, my data does not come as handy csv files but as

Bucket pruning

Hi, We created a bucketed table and when we select in the following way: select * from testtble where bucket_col ='X'; We observe that there all of the table is being read and not just the specific bucket. Does Hive support such a feature ? Thanks, Daniel

Trying to improve compression ratio for an ORC table

Hi guys, I'm experiencing something very odd: I have an ORC table with the orc.compress=SNAPPY property that weighs 4.9 GB and is composed of 253 files.. I then do a CTAS into a new table where I added this property orc.compress.size=2485760 to improve the compression ratio. The new table weighs

Re: Adding new columns to parquet based Hive table

Hi Kumar, Altering the table just update's Hive's metadata without updating parquet's schema. I believe that if you'll insert to your table (after adding the column) you'll be able to later on select all 3 columns. Daniel On 14 בינו׳ 2015, at 21:34, Kumar V kumarbuyonl...@yahoo.com wrote:

Monitoring Hive Thread Usage

Hi, I suspect we have a problem with clients opening connections and not closing them. To verify that I'd like to monitor the Hive's number of threads but I can't seem to find a way to do so. Anyone has ever tried or has any ideas? Thanks, Daniel

Re: Monitoring Hive Thread Usage

Found a solution (aside from JMX): ps -eLf | grep [HiveServer2 PID] On Tue, Jan 6, 2015 at 11:03 AM, Daniel Haviv daniel.ha...@veracity-group.com wrote: Hi, I suspect we have a problem with clients opening connections and not closing them. To verify that I'd like to monitor the Hive's

Re: How to pass information to hive udf except as arguments

First result in google: http://stackoverflow.com/questions/12464636/how-to-set-variables-in-hive-scripts Daniel On 19 בדצמ׳ 2014, at 10:54, Dilip Agarwal dilip.jindal1...@gmail.com wrote: Hi, I have created a udf which accepts geo location points as arguments and return the name of

Re: Case inside select statement in hive

Hi, Please RTFM before asking questions. Taken from https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF: Conditional Functions *Return Type* *Name(Signature)* *Description* T if(boolean testCondition, T valueTrue, T valueFalseOrNull) Returns valueTrue when testCondition is

Re: Concatenating ORC files

at 3:29 AM, Daniel Haviv daniel.ha...@veracity-group.com wrote: Hi, I've created a JIRA with a test case: https://issues.apache.org/jira/browse/HIVE-9080 Thanks! Daniel On Thu, Dec 11, 2014 at 12:49 AM, Prasanth Jayachandran pjayachand...@hortonworks.com wrote: I am unable to reproduce

Re: Concatenating ORC files

not sure why does it fail in case 1 though. Can you create a jira with a reproducible case? I can take a look at it. - Prasanth On Wed, Dec 10, 2014 at 10:37 AM, Daniel Haviv daniel.ha...@veracity-group.com wrote: I've made a little experiment and recreated the table with 'orc.create.index

Re: Concatenating ORC files

(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) It seems that the concatenation feature needs more work.. Daniel On Wed, Dec 10, 2014 at 4:54 PM, Daniel Haviv daniel.ha

Insert into dynamic partitions performance

Hi, I'm executing an insert statement that goes over 1TB of data. The map phase goes well but the reduce stage only used one reducer which becomes a great bottleneck. I've tried to set the number of reducers to four and added a distribute by clause to the statement but I'm still using just one

Re: Insert into dynamic partitions performance

Thanks Gopal, I dont want to divide my data any further. Isn't there a way to make hive allocate more than one reducer for the whole job? Maybe one per partition. Daniel On 7 בדצמ׳ 2014, at 06:06, Gopal V gop...@apache.org wrote: On 12/6/14, 6:27 AM, Daniel Haviv wrote: Hi, I'm executing

Re: Insert into dynamic partitions performance

I see. Thanks a lot that's very helpful! Daniel On 7 בדצמ׳ 2014, at 09:10, Gopal V gop...@apache.org wrote: On 12/6/14, 10:11 PM, Daniel Haviv wrote: Isn't there a way to make hive allocate more than one reducer for the whole job? Maybe one per partition. Yes

Re: Start hiveserver2 as a daemon

Try using screen Daniel On 5 בדצמ׳ 2014, at 19:08, peterm_second regest...@gmail.com wrote: yes, I've tried nohup , even sh -c . works but after the first call get's executed in the background I get the message you can see when a hadoop job is submitted to the cluster and then the

Running hive inside a bash script

Hi, I have a bash script that runs a hive query and I would like it to do something if the query succeeds and something else if it fails. My testings show that a query failure does not change Hive's exit code, what's the right way to achieve this ? Thanks, Daniel

Re: Container launch failed Error

It seems that the application master can't resolve slave6's name to an IP Daniel On 24 בנוב׳ 2014, at 18:49, Amit Behera amit.bd...@gmail.com wrote: Hi Users, my cluster(1+8) configuration: RAM : 32 GB each HDFS : 1.5 TB SSD CPU : 8 core each

Re: Container launch failed Error

? But for small data (up to 20 GB table) it is running and for 300GB table only count(*) running sometimes and sometimes failed Thanks Amit On Mon, Nov 24, 2014 at 10:37 PM, Daniel Haviv daniel.ha...@veracity-group.com wrote: did you copy the hosts file to all the nodes? Daniel

Re: Container launch failed Error

Good luck Share your results with us Daniel On 24 בנוב׳ 2014, at 19:36, Amit Behera amit.bd...@gmail.com wrote: Hi Daniel, Thanks a lot, I will do that and rerun the query. :) On Mon, Nov 24, 2014 at 10:59 PM, Daniel Haviv daniel.ha...@veracity-group.com wrote: It is a problem

Problem after upgrading to hive 0.14

Hi, After upgrading to hive 0.14 any query I run I hit the following message: . . . . . . . . . . . . . . . . ; INFO : Tez session hasn't been created yet. Opening session Error: Error while processing statement: FAILED: Execution Error, return code -101 from