[jira] [Updated] (HIVE-13977) nvl funtion not working after left outer join
[ https://issues.apache.org/jira/browse/HIVE-13977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] balaswamy vaddeman updated HIVE-13977: -- Description: Recreating problem. 1).Create table with sample data. create table tabletest (n bigint, t string); insert into tabletest values (1, 'one'); insert into tabletest values(2, 'two'); 2) Run leftouter join query on single table. select a.n as leftHandN , b.n as rightHandN , b.t as rightHandT , nvl(b.t,"empty") as rightHandTnvl -- Expected empty --> received empty , nvl(b.n,-1) as rightHandNnvl -- Expected -1 --> received 1 from ( select * from tabletest where n=1 ) a left outer join ( select * from tabletest where 1=2 ) b on a.n = b.n; nvl(b.n,-1) should return -1 but returns 1. I have found b.n always returning a.n value.if a.n is 1 ,b.n is returning 1 and if it is 2,same 2 will be returned. More information: length(b.n) ---gives--->1 cast(b.n as string) --gives--->1 ascii(b.n) --gives>49 i.e 1 was: Recreating problem. 1).Create table with sample data. create table tabletest (n bigint, t string); insert into tabletest values (1, 'one'); insert into tabletest values(2, 'two'); 2) Run leftouter join query on single table. select a.n as leftHandN , b.n as rightHandN , b.t as rightHandT , nvl(b.t,"empty") as rightHandTnvl -- Expected empty --> received empty , nvl(b.n,-1) as rightHandNnvl -- Expected -1 --> received 1 from ( select * from tabletest where n=1 ) a left outer join ( select * from tabletest where 1=2 ) b on a.n = b.n; nvl(b.n,-1) should return -1 but returns 1. I have found b.n always returning a.n value.if a.n is 1 ,b.n is returning 1 and if it is 2,same 2 will be returned. > nvl funtion not working after left outer join > -- > > Key: HIVE-13977 > URL: https://issues.apache.org/jira/browse/HIVE-13977 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1 >Reporter: balaswamy vaddeman > > Recreating problem. > 1).Create table with sample data. > create table tabletest (n bigint, t string); > insert into tabletest values (1, 'one'); > insert into tabletest values(2, 'two'); > 2) Run leftouter join query on single table. > select a.n as leftHandN > , b.n as rightHandN > , b.t as rightHandT > , nvl(b.t,"empty") as rightHandTnvl -- Expected empty --> received empty > , nvl(b.n,-1) as rightHandNnvl -- Expected -1 --> received 1 > from > ( > select * > from tabletest > where n=1 > ) a > left outer join > ( > select * > from tabletest > where 1=2 > ) b > on a.n = b.n; > nvl(b.n,-1) should return -1 but returns 1. > I have found b.n always returning a.n value.if a.n is 1 ,b.n is returning 1 > and if it is 2,same 2 will be returned. > More information: > length(b.n) ---gives--->1 > cast(b.n as string) --gives--->1 > ascii(b.n) --gives>49 i.e 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13977) nvl funtion not working after left outer join
[
https://issues.apache.org/jira/browse/HIVE-13977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323915#comment-15323915
]
balaswamy vaddeman commented on HIVE-13977:
---
Plan not optimized by CBO.
Vertex dependency in root stage
Map 1 <- Map 2 (BROADCAST_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Map 1
File Output Operator [FS_10]
compressed:false
Statistics:Num rows: 1 Data size: 5 Basic stats: COMPLETE Column
stats: NONE
table:{"input
format:":"org.apache.hadoop.mapred.TextInputFormat","output
format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_9]
outputColumnNames:["_col0","_col1","_col2","_col3","_col4"]
Statistics:Num rows: 1 Data size: 5 Basic stats: COMPLETE Column
stats: NONE
Map Join Operator [MAPJOIN_14]
| condition map:[{"":"Left Outer Join0 to 1"}]
| HybridGraceHashJoin:true
| keys:{"Map 2":"_col0 (type: bigint)","Map 1":"_col0 (type:
bigint)"}
| outputColumnNames:["_col0","_col2","_col3"]
| Statistics:Num rows: 1 Data size: 5 Basic stats: COMPLETE
Column stats: NONE
|<-Map 2 [BROADCAST_EDGE]
| Reduce Output Operator [RS_7]
| key expressions:_col0 (type: bigint)
| Map-reduce partition columns:_col0 (type: bigint)
| sort order:+
| Statistics:Num rows: 1 Data size: 5 Basic stats: COMPLETE
Column stats: NONE
| value expressions:_col1 (type: string)
| Select Operator [SEL_5]
|outputColumnNames:["_col0","_col1"]
|Statistics:Num rows: 1 Data size: 5 Basic stats:
COMPLETE Column stats: NONE
|Filter Operator [FIL_13]
| predicate:false (type: boolean)
| Statistics:Num rows: 1 Data size: 5 Basic stats:
COMPLETE Column stats: NONE
| TableScan [TS_3]
| alias:tabletest
| Statistics:Num rows: 2 Data size: 10 Basic stats:
COMPLETE Column stats: NONE
|<-Select Operator [SEL_2]
outputColumnNames:["_col0"]
Statistics:Num rows: 1 Data size: 5 Basic stats: COMPLETE
Column stats: NONE
Filter Operator [FIL_12]
predicate:(n = 1) (type: boolean)
Statistics:Num rows: 1 Data size: 5 Basic stats:
COMPLETE Column stats: NONE
TableScan [TS_0]
alias:tabletest
Statistics:Num rows: 2 Data size: 10 Basic stats:
COMPLETE Column stats: NONE
Time taken: 0.196 seconds, Fetched: 49 row(s)
> nvl funtion not working after left outer join
> --
>
> Key: HIVE-13977
> URL: https://issues.apache.org/jira/browse/HIVE-13977
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: balaswamy vaddeman
>
> Recreating problem.
> 1).Create table with sample data.
> create table tabletest (n bigint, t string);
> insert into tabletest values (1, 'one');
> insert into tabletest values(2, 'two');
> 2) Run leftouter join query on single table.
> select a.n as leftHandN
> , b.n as rightHandN
> , b.t as rightHandT
> , nvl(b.t,"empty") as rightHandTnvl -- Expected empty --> received empty
> , nvl(b.n,-1) as rightHandNnvl -- Expected -1 --> received 1
> from
> (
> select *
> from tabletest
> where n=1
> ) a
> left outer join
> (
> select *
> from tabletest
> where 1=2
> ) b
> on a.n = b.n;
> nvl(b.n,-1) should return -1 but returns 1.
> I have found b.n always returning a.n value.if a.n is 1 ,b.n is returning 1
> and if it is 2,same 2 will be returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13977) nvl funtion not working after left outer join
[ https://issues.apache.org/jira/browse/HIVE-13977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] balaswamy vaddeman updated HIVE-13977: -- Description: Recreating problem. 1).Create table with sample data. create table tabletest (n bigint, t string); insert into tabletest values (1, 'one'); insert into tabletest values(2, 'two'); 2) Run leftouter join query on single table. select a.n as leftHandN , b.n as rightHandN , b.t as rightHandT , nvl(b.t,"empty") as rightHandTnvl -- Expected empty --> received empty , nvl(b.n,-1) as rightHandNnvl -- Expected -1 --> received 1 from ( select * from tabletest where n=1 ) a left outer join ( select * from tabletest where 1=2 ) b on a.n = b.n; nvl(b.n,-1) should return -1 but returns 1. I have found b.n always returning a.n value.if a.n is 1 ,b.n is returning 1 and if it is 2,same 2 will be returned. was: Recreating problem. 1).Create table with sample data. create table tabletest (n bigint, t string); insert into tabletest values (1, 'one'); insert into tabletest values(2, 'two'); 2) Run leftouter join query on single table. select a.n as leftHandN , b.n as rightHandN , b.t as rightHandT , nvl(b.t,"empty") as rightHandTnvl -- Expected empty --> received empty , nvl(b.n,-1) as rightHandNnvl -- Expected -1 --> received 1 from ( select * from tabletest where n=1 ) a left outer join ( select * from tabletest where 1=2 ) on a.n = b.n; nvl(b.n,-1) should return -1 but returns 1. I have found b.n always returning a.n value.if a.n is 1 ,b.n is returning 1 and if it is 2,same 2 will be returned. > nvl funtion not working after left outer join > -- > > Key: HIVE-13977 > URL: https://issues.apache.org/jira/browse/HIVE-13977 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1 >Reporter: balaswamy vaddeman > > Recreating problem. > 1).Create table with sample data. > create table tabletest (n bigint, t string); > insert into tabletest values (1, 'one'); > insert into tabletest values(2, 'two'); > 2) Run leftouter join query on single table. > select a.n as leftHandN > , b.n as rightHandN > , b.t as rightHandT > , nvl(b.t,"empty") as rightHandTnvl -- Expected empty --> received empty > , nvl(b.n,-1) as rightHandNnvl -- Expected -1 --> received 1 > from > ( > select * > from tabletest > where n=1 > ) a > left outer join > ( > select * > from tabletest > where 1=2 > ) b > on a.n = b.n; > nvl(b.n,-1) should return -1 but returns 1. > I have found b.n always returning a.n value.if a.n is 1 ,b.n is returning 1 > and if it is 2,same 2 will be returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian updated HIVE-13432: -- Attachment: orc_hive-site.xml > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > Attachments: orc_hive-site.xml > > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian updated HIVE-13432: -- Attachment: (was: orc_hive-site.xml) > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiuzhuang Lian updated HIVE-13432: -- Attachment: orc_hive-site.xml hive-site.xml to pr-produce the compaction failure for HIVE-13432 > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > Attachments: orc_hive-site.xml > > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323902#comment-15323902 ] Qiuzhuang Lian commented on HIVE-13432: --- Sorry for delay response. Yes, our hive-site.xml uses CombineHiveInputFormat. I should have sent you our hive-site.xml. Please check the attached to see if any more clue. > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13803) More aggressive inference of transitive predicates for inner joins
[
https://issues.apache.org/jira/browse/HIVE-13803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323868#comment-15323868
]
Hive QA commented on HIVE-13803:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809138/HIVE-13803.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 32 failed/errored test(s), 10223 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_subq_not_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_not_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_join_preds
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_not_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_avro_joins_native
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_subq_not_in
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby7_map_skew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join39
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_join_union
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoinopt15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoinopt17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_exists
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_11
org.apache.hadoop.hive.ql.TestTxnCommands.testSimpleAcidInsert
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/68/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/68/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-68/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 32 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12809138 - PreCommit-HIVE-MASTER-Build
> More aggressive inference of transitive predicates for inner joins
> --
>
> Key: HIVE-13803
> URL: https://issues.apache.org/jira/browse/HIVE-13803
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13803.01.patch, HIVE-13803.patch
>
>
> Follow-up of HIVE-13068.
> Currently for inner joins, we do not infer transitive predicates that do not
> reference any of the columns of the input. These predicates can be evaluated
> statically and can be useful to quickly discard intermediary results.
> Ex. ql/src/test/results/clientpositive/constprog3.q.out
> {noformat}
> explain
> select table1.id, table1.val, table1.val1
> from table1 inner join table3
> on table1.dimid = table3.id and table3.id = 1 where table1.dimid <> 1
> {noformat}
> Current plan:
> {noformat}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: table1
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
[jira] [Updated] (HIVE-13960) Session timeout may happen before HIVE_SERVER2_IDLE_SESSION_TIMEOUT for back-to-back synchronous operations.
[
https://issues.apache.org/jira/browse/HIVE-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HIVE-13960:
-
Status: Patch Available (was: Open)
> Session timeout may happen before HIVE_SERVER2_IDLE_SESSION_TIMEOUT for
> back-to-back synchronous operations.
>
>
> Key: HIVE-13960
> URL: https://issues.apache.org/jira/browse/HIVE-13960
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HIVE-13960.000.patch
>
>
> Session timeout may happen before
> HIVE_SERVER2_IDLE_SESSION_TIMEOUT(hive.server2.idle.session.timeout) for
> back-to-back synchronous operations.
> This issue can happen with the following two operations op1 and op2: op2 is a
> synchronous long running operation, op2 is running right after op1 is closed.
>
> 1. closeOperation(op1) is called:
> this will set {{lastIdleTime}} with value System.currentTimeMillis() because
> {{opHandleSet}} becomes empty after {{closeOperation}} remove op1 from
> {{opHandleSet}}.
> 2. op2 is running for long time by calling {{executeStatement}} right after
> closeOperation(op1) is called.
> If op2 is running for more than HIVE_SERVER2_IDLE_SESSION_TIMEOUT, then the
> session will timeout even when op2 is still running.
> We hit this issue when we use PyHive to execute non-async operation
> The following is the exception we see:
> {code}
> File "/usr/local/lib/python2.7/dist-packages/pyhive/hive.py", line 126, in
> close
> _check_status(response)
> File "/usr/local/lib/python2.7/dist-packages/pyhive/hive.py", line 362, in
> _check_status
> raise OperationalError(response)
> OperationalError: TCloseSessionResp(status=TStatus(errorCode=0,
> errorMessage='Session does not exist!', sqlState=None,
> infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Session does not
> exist!:12:11',
> 'org.apache.hive.service.cli.session.SessionManager:closeSession:SessionManager.java:311',
> 'org.apache.hive.service.cli.CLIService:closeSession:CLIService.java:221',
> 'org.apache.hive.service.cli.thrift.ThriftCLIService:CloseSession:ThriftCLIService.java:471',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1273',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1258',
> 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39',
> 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39',
> 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56',
>
> 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285',
>
> 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145',
>
> 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615',
> 'java.lang.Thread:run:Thread.java:745'], statusCode=3))
> TCloseSessionResp(status=TStatus(errorCode=0, errorMessage='Session does not
> exist!', sqlState=None,
> infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Session does not
> exist!:12:11',
> 'org.apache.hive.service.cli.session.SessionManager:closeSession:SessionManager.java:311',
> 'org.apache.hive.service.cli.CLIService:closeSession:CLIService.java:221',
> 'org.apache.hive.service.cli.thrift.ThriftCLIService:CloseSession:ThriftCLIService.java:471',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1273',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1258',
> 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39',
> 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39',
> 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56',
>
> 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285',
>
> 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145',
>
> 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615',
> 'java.lang.Thread:run:Thread.java:745'], statusCode=3))
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13960) Session timeout may happen before HIVE_SERVER2_IDLE_SESSION_TIMEOUT for back-to-back synchronous operations.
[
https://issues.apache.org/jira/browse/HIVE-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HIVE-13960:
-
Status: Open (was: Patch Available)
> Session timeout may happen before HIVE_SERVER2_IDLE_SESSION_TIMEOUT for
> back-to-back synchronous operations.
>
>
> Key: HIVE-13960
> URL: https://issues.apache.org/jira/browse/HIVE-13960
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HIVE-13960.000.patch
>
>
> Session timeout may happen before
> HIVE_SERVER2_IDLE_SESSION_TIMEOUT(hive.server2.idle.session.timeout) for
> back-to-back synchronous operations.
> This issue can happen with the following two operations op1 and op2: op2 is a
> synchronous long running operation, op2 is running right after op1 is closed.
>
> 1. closeOperation(op1) is called:
> this will set {{lastIdleTime}} with value System.currentTimeMillis() because
> {{opHandleSet}} becomes empty after {{closeOperation}} remove op1 from
> {{opHandleSet}}.
> 2. op2 is running for long time by calling {{executeStatement}} right after
> closeOperation(op1) is called.
> If op2 is running for more than HIVE_SERVER2_IDLE_SESSION_TIMEOUT, then the
> session will timeout even when op2 is still running.
> We hit this issue when we use PyHive to execute non-async operation
> The following is the exception we see:
> {code}
> File "/usr/local/lib/python2.7/dist-packages/pyhive/hive.py", line 126, in
> close
> _check_status(response)
> File "/usr/local/lib/python2.7/dist-packages/pyhive/hive.py", line 362, in
> _check_status
> raise OperationalError(response)
> OperationalError: TCloseSessionResp(status=TStatus(errorCode=0,
> errorMessage='Session does not exist!', sqlState=None,
> infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Session does not
> exist!:12:11',
> 'org.apache.hive.service.cli.session.SessionManager:closeSession:SessionManager.java:311',
> 'org.apache.hive.service.cli.CLIService:closeSession:CLIService.java:221',
> 'org.apache.hive.service.cli.thrift.ThriftCLIService:CloseSession:ThriftCLIService.java:471',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1273',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1258',
> 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39',
> 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39',
> 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56',
>
> 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285',
>
> 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145',
>
> 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615',
> 'java.lang.Thread:run:Thread.java:745'], statusCode=3))
> TCloseSessionResp(status=TStatus(errorCode=0, errorMessage='Session does not
> exist!', sqlState=None,
> infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Session does not
> exist!:12:11',
> 'org.apache.hive.service.cli.session.SessionManager:closeSession:SessionManager.java:311',
> 'org.apache.hive.service.cli.CLIService:closeSession:CLIService.java:221',
> 'org.apache.hive.service.cli.thrift.ThriftCLIService:CloseSession:ThriftCLIService.java:471',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1273',
>
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1258',
> 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39',
> 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39',
> 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56',
>
> 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285',
>
> 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145',
>
> 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615',
> 'java.lang.Thread:run:Thread.java:745'], statusCode=3))
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13914) Changing a column of orc table from float to double adds some fractional digits
[
https://issues.apache.org/jira/browse/HIVE-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline resolved HIVE-13914.
-
Resolution: Not A Problem
"Works as Designed"
> Changing a column of orc table from float to double adds some fractional
> digits
> ---
>
> Key: HIVE-13914
> URL: https://issues.apache.org/jira/browse/HIVE-13914
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>
> Some fractional digits are added after altering a float column of an orc
> table to double:
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table test;
> No rows affected (0.233 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (f float)
> stored as orc;
> No rows affected (0.095 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test
> values(-35664.76),(29497.34);
> No rows affected (2.206 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.131 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> alter table test change column
> f f double;
> No rows affected (0.183 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +--+--+
> | test.f |
> +--+--+
> | -35664.76171875 |
> | 29497.33984375 |
> +--+--+
> 2 rows selected (0.114 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> {noformat}
> The issue is NOT seen with text formatted table:
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (f float) ;
> No rows affected (0.131 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test
> values(-35664.76),(29497.34);
> No rows affected (2.114 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.163 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> alter table test change column
> f f double;
> No rows affected (0.184 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.11 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13914) Changing a column of orc table from float to double adds some fractional digits
[
https://issues.apache.org/jira/browse/HIVE-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323795#comment-15323795
]
Matt McCline commented on HIVE-13914:
-
think this is part of the float madness.
I'm concluding that TEXTFILE format tables are not a good gold standard for
what needs to be produced.
For Schema Evolution, I think that when there is a data type conversion the
gold standard for ORC is CAST.
If you take the TEXTFILE table with float data type and do a "select cast(f as
double) from test_text;" you will get:
-35664.76171875
29497.33984375
but after the column is changed to double a plain select * will produce:
-35664.76
29497.34
And, because there is no history that the file was written as float there is no
hope of fixing this problem.
> Changing a column of orc table from float to double adds some fractional
> digits
> ---
>
> Key: HIVE-13914
> URL: https://issues.apache.org/jira/browse/HIVE-13914
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>
> Some fractional digits are added after altering a float column of an orc
> table to double:
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table test;
> No rows affected (0.233 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (f float)
> stored as orc;
> No rows affected (0.095 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test
> values(-35664.76),(29497.34);
> No rows affected (2.206 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.131 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> alter table test change column
> f f double;
> No rows affected (0.183 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +--+--+
> | test.f |
> +--+--+
> | -35664.76171875 |
> | 29497.33984375 |
> +--+--+
> 2 rows selected (0.114 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> {noformat}
> The issue is NOT seen with text formatted table:
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (f float) ;
> No rows affected (0.131 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test
> values(-35664.76),(29497.34);
> No rows affected (2.114 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.163 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> alter table test change column
> f f double;
> No rows affected (0.184 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.11 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13914) Changing a column of orc table from float to double adds some fractional digits
[
https://issues.apache.org/jira/browse/HIVE-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323794#comment-15323794
]
Matt McCline commented on HIVE-13914:
-
(Comment from Gunther):
I've tried it in java and found the same behavior.
{noformat}
float f = 29497.34f;
System.out.println(f);
System.out.println((double)f);
{noformat}
yields
{noformat}
29497.34
29497.33984375
{noformat}
On the other hand I'm pretty sure 29497.34 can be represented as double (as
happens with text).
> Changing a column of orc table from float to double adds some fractional
> digits
> ---
>
> Key: HIVE-13914
> URL: https://issues.apache.org/jira/browse/HIVE-13914
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>
> Some fractional digits are added after altering a float column of an orc
> table to double:
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table test;
> No rows affected (0.233 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (f float)
> stored as orc;
> No rows affected (0.095 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test
> values(-35664.76),(29497.34);
> No rows affected (2.206 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.131 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> alter table test change column
> f f double;
> No rows affected (0.183 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +--+--+
> | test.f |
> +--+--+
> | -35664.76171875 |
> | 29497.33984375 |
> +--+--+
> 2 rows selected (0.114 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> {noformat}
> The issue is NOT seen with text formatted table:
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (f float) ;
> No rows affected (0.131 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test
> values(-35664.76),(29497.34);
> No rows affected (2.114 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.163 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> alter table test change column
> f f double;
> No rows affected (0.184 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.11 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13914) Changing a column of orc table from float to double adds some fractional digits
[
https://issues.apache.org/jira/browse/HIVE-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323793#comment-15323793
]
Matt McCline commented on HIVE-13914:
-
ORC
The FloatTreeReader.nextVector reads the first float as:
-35664.76
and stores it into the DoubleColumnVector.vector (double[]) where it displays
as:
-35664.76171875
And, FloatTreeReader.nextVector reads the second float as:
29497.34
and stores it into the DoubleColumnVector.vector (double[]) where it displays
as:
29497.33984375
For the query looking at the original data type float, the
RecordReaderImpl.nextFloat casts the DoubleColumnVector.vector entry to float
and sets the FloatWritable. The values -35664.76 and 29497.34 reappear.
For the 2nd query after changing the data type to double, the
RecordReaderImpl.nextDouble just stores the DoubleColumnVector.vector entry in
the DoubleWritable. The values with more digits -35664.76171875 and
29497.33984375 are returned.
--
TEXTFILE
The float values are stored as text fields "-35664.76" and "29497.34" and there
is no conversion because we are using a table. There is just a new
interpretation of the field.
The first query reads those fields as float and the second query after changing
the data type to double reads the text fields as double.
> Changing a column of orc table from float to double adds some fractional
> digits
> ---
>
> Key: HIVE-13914
> URL: https://issues.apache.org/jira/browse/HIVE-13914
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>
> Some fractional digits are added after altering a float column of an orc
> table to double:
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table test;
> No rows affected (0.233 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (f float)
> stored as orc;
> No rows affected (0.095 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test
> values(-35664.76),(29497.34);
> No rows affected (2.206 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.131 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> alter table test change column
> f f double;
> No rows affected (0.183 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +--+--+
> | test.f |
> +--+--+
> | -35664.76171875 |
> | 29497.33984375 |
> +--+--+
> 2 rows selected (0.114 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> {noformat}
> The issue is NOT seen with text formatted table:
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (f float) ;
> No rows affected (0.131 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test
> values(-35664.76),(29497.34);
> No rows affected (2.114 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.163 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> alter table test change column
> f f double;
> No rows affected (0.184 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.11 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13851) Change column of textfile formatted table from float to double will drop some fractional digits
[
https://issues.apache.org/jira/browse/HIVE-13851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline resolved HIVE-13851.
-
Resolution: Not A Problem
"Works As Designed"
> Change column of textfile formatted table from float to double will drop some
> fractional digits
> ---
>
> Key: HIVE-13851
> URL: https://issues.apache.org/jira/browse/HIVE-13851
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>
> In a case of texfile formatted table, changing a column from float to double
> will drop some fractional digits:
> {noformat}
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> create table test(f float);
> No rows affected (0.176 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> insert into table test
> values(-35664.76171875),(29497.349609375);
> INFO : Tez session hasn't been created yet. Opening session
> INFO : Dag name: insert into table tes...5),(29497.349609375)(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464135801747_0400)
> INFO : Map 1: -/-
> INFO : Map 1: 0/1
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test from
> hdfs://os-r7-wbucju-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-05-25_17-22-03_941_5459171987449455387-1/-ext-1
> INFO : Table default.test stats: [numFiles=1, numRows=2, totalSize=19,
> rawDataSize=17]
> No rows affected (18.443 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from test;
> +--+--+
> | test.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.231 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> alter table test change column
> f f double;
> No rows affected (0.34 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.35 |
> ++--+
> 2 rows selected (0.128 seconds)
> {noformat}
> The issue is not seen with orc table:
> {noformat}
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> create table orc(f float)
> stored as orc;
> No rows affected (0.162 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> insert into table orc
> values(-35664.76171875),(29497.349609375);
> INFO : Session is already open
> INFO : Dag name: insert into table orc...5),(29497.349609375)(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464135801747_0400)
> INFO : Map 1: 0/1
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.orc from
> hdfs://os-r7-wbucju-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/orc/.hive-staging_hive_2016-05-25_17-25-26_569_221649193772012279-1/-ext-1
> INFO : Table default.orc stats: [numFiles=1, numRows=2, totalSize=253,
> rawDataSize=8]
> No rows affected (7.87 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from orc;
> +--+--+
> | orc.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.117 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> alter table orc change column
> f f double;
> No rows affected (0.85 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from orc;
> +--+--+
> | orc.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.168 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13851) Change column of textfile formatted table from float to double will drop some fractional digits
[
https://issues.apache.org/jira/browse/HIVE-13851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323790#comment-15323790
]
Matt McCline commented on HIVE-13851:
-
In UDFToFloat used by the INSERT VALUES to convert the VALUES clause text to
float (or double):
float f = Float.valueOf("-35664.76171875") --> -35664.76
double d = Double.valueOf("-35664.76171875") --> -35664.76171875
> Change column of textfile formatted table from float to double will drop some
> fractional digits
> ---
>
> Key: HIVE-13851
> URL: https://issues.apache.org/jira/browse/HIVE-13851
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>
> In a case of texfile formatted table, changing a column from float to double
> will drop some fractional digits:
> {noformat}
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> create table test(f float);
> No rows affected (0.176 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> insert into table test
> values(-35664.76171875),(29497.349609375);
> INFO : Tez session hasn't been created yet. Opening session
> INFO : Dag name: insert into table tes...5),(29497.349609375)(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464135801747_0400)
> INFO : Map 1: -/-
> INFO : Map 1: 0/1
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test from
> hdfs://os-r7-wbucju-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-05-25_17-22-03_941_5459171987449455387-1/-ext-1
> INFO : Table default.test stats: [numFiles=1, numRows=2, totalSize=19,
> rawDataSize=17]
> No rows affected (18.443 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from test;
> +--+--+
> | test.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.231 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> alter table test change column
> f f double;
> No rows affected (0.34 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.35 |
> ++--+
> 2 rows selected (0.128 seconds)
> {noformat}
> The issue is not seen with orc table:
> {noformat}
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> create table orc(f float)
> stored as orc;
> No rows affected (0.162 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> insert into table orc
> values(-35664.76171875),(29497.349609375);
> INFO : Session is already open
> INFO : Dag name: insert into table orc...5),(29497.349609375)(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464135801747_0400)
> INFO : Map 1: 0/1
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.orc from
> hdfs://os-r7-wbucju-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/orc/.hive-staging_hive_2016-05-25_17-25-26_569_221649193772012279-1/-ext-1
> INFO : Table default.orc stats: [numFiles=1, numRows=2, totalSize=253,
> rawDataSize=8]
> No rows affected (7.87 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from orc;
> +--+--+
> | orc.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.117 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> alter table orc change column
> f f double;
> No rows affected (0.85 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from orc;
> +--+--+
> | orc.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.168 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13851) Change column of textfile formatted table from float to double will drop some fractional digits
[
https://issues.apache.org/jira/browse/HIVE-13851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323789#comment-15323789
]
Matt McCline commented on HIVE-13851:
-
For the TEXTFILE case, if you convert the float column to string (as a way of
seeing what is being stored in the text field!) instead of double it reveals
the following digit clipped values are what was stored:
-35664.76
29497.35
Interesting.
Now, if you create a input file and insert data with "load data local inpath
", you get the expected result:
-35664.76171875
29497.349609375
So, that leads to why are values specified in the INSERT VALUES clause
different than data in an input file?
Perhaps they being parsed as decimal with scale 2 and then being converted to
string and stored in the temporary file that serves as input to the INSERT
statement?
> Change column of textfile formatted table from float to double will drop some
> fractional digits
> ---
>
> Key: HIVE-13851
> URL: https://issues.apache.org/jira/browse/HIVE-13851
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>
> In a case of texfile formatted table, changing a column from float to double
> will drop some fractional digits:
> {noformat}
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> create table test(f float);
> No rows affected (0.176 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> insert into table test
> values(-35664.76171875),(29497.349609375);
> INFO : Tez session hasn't been created yet. Opening session
> INFO : Dag name: insert into table tes...5),(29497.349609375)(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464135801747_0400)
> INFO : Map 1: -/-
> INFO : Map 1: 0/1
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test from
> hdfs://os-r7-wbucju-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-05-25_17-22-03_941_5459171987449455387-1/-ext-1
> INFO : Table default.test stats: [numFiles=1, numRows=2, totalSize=19,
> rawDataSize=17]
> No rows affected (18.443 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from test;
> +--+--+
> | test.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.231 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> alter table test change column
> f f double;
> No rows affected (0.34 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.35 |
> ++--+
> 2 rows selected (0.128 seconds)
> {noformat}
> The issue is not seen with orc table:
> {noformat}
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> create table orc(f float)
> stored as orc;
> No rows affected (0.162 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> insert into table orc
> values(-35664.76171875),(29497.349609375);
> INFO : Session is already open
> INFO : Dag name: insert into table orc...5),(29497.349609375)(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464135801747_0400)
> INFO : Map 1: 0/1
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.orc from
> hdfs://os-r7-wbucju-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/orc/.hive-staging_hive_2016-05-25_17-25-26_569_221649193772012279-1/-ext-1
> INFO : Table default.orc stats: [numFiles=1, numRows=2, totalSize=253,
> rawDataSize=8]
> No rows affected (7.87 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from orc;
> +--+--+
> | orc.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.117 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> alter table orc change column
> f f double;
> No rows affected (0.85 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from orc;
> +--+--+
> | orc.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.168 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13847) Avoid file open call in RecordReaderUtils as the stream is already available
[ https://issues.apache.org/jira/browse/HIVE-13847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323785#comment-15323785 ] Ashutosh Chauhan commented on HIVE-13847: - [~rajesh.balamohan] What do you think of [~prasanth_j] comment? > Avoid file open call in RecordReaderUtils as the stream is already available > > > Key: HIVE-13847 > URL: https://issues.apache.org/jira/browse/HIVE-13847 > Project: Hive > Issue Type: Improvement > Components: ORC >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13847.1.patch > > > File open call in RecordReaderUtils::readRowIndex can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13617) LLAP: support non-vectorized execution in IO
[
https://issues.apache.org/jira/browse/HIVE-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323774#comment-15323774
]
Hive QA commented on HIVE-13617:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809101/HIVE-13617.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10224 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_llap_nonvector
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_llapdecider
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/67/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/67/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-67/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12809101 - PreCommit-HIVE-MASTER-Build
> LLAP: support non-vectorized execution in IO
>
>
> Key: HIVE-13617
> URL: https://issues.apache.org/jira/browse/HIVE-13617
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13617-wo-11417.patch, HIVE-13617-wo-11417.patch,
> HIVE-13617.01.patch, HIVE-13617.03.patch, HIVE-13617.04.patch,
> HIVE-13617.05.patch, HIVE-13617.patch, HIVE-13617.patch,
> HIVE-15396-with-oi.patch
>
>
> Two approaches - a separate decoding path, into rows instead of VRBs; or
> decoding VRBs into rows on a higher level (the original LlapInputFormat). I
> think the latter might be better - it's not a hugely important path, and perf
> in non-vectorized case is not the best anyway, so it's better to make do with
> much less new code and architectural disruption.
> Some ORC patches in progress introduce an easy to reuse (or so I hope,
> anyway) VRB-to-row conversion, so we should just use that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13991) Union All on view fail with no valid permission on underneath table
[
https://issues.apache.org/jira/browse/HIVE-13991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13991:
Status: Patch Available (was: Open)
Need code review.
> Union All on view fail with no valid permission on underneath table
> ---
>
> Key: HIVE-13991
> URL: https://issues.apache.org/jira/browse/HIVE-13991
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13991.1.patch
>
>
> When sentry is enabled.
> create view V as select * from T;
> When the user has read permission on view V, but does not have read
> permission on table T,
> select * from V union all select * from V
> failed with:
> {noformat}
> 0: jdbc:hive2://> select * from s07view union all select * from
> s07view limit 1;
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privileges for this query:
> Server=server1->Db=default->Table=sample_07->action=select;
> (state=42000,code=4)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13991) Union All on view fail with no valid permission on underneath table
[
https://issues.apache.org/jira/browse/HIVE-13991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13991:
Attachment: HIVE-13991.1.patch
> Union All on view fail with no valid permission on underneath table
> ---
>
> Key: HIVE-13991
> URL: https://issues.apache.org/jira/browse/HIVE-13991
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13991.1.patch
>
>
> When sentry is enabled.
> create view V as select * from T;
> When the user has read permission on view V, but does not have read
> permission on table T,
> select * from V union all select * from V
> failed with:
> {noformat}
> 0: jdbc:hive2://> select * from s07view union all select * from
> s07view limit 1;
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privileges for this query:
> Server=server1->Db=default->Table=sample_07->action=select;
> (state=42000,code=4)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13991) Union All on view fail with no valid permission on underneath table
[
https://issues.apache.org/jira/browse/HIVE-13991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323765#comment-15323765
]
Yongzhi Chen commented on HIVE-13991:
-
There are inconsistent key values between put and get operations for map
viewAliasToInput
In
private void getMetaData(QB qb, ReadEntity parentInput) method:
{noformat}
// This is the last time we'll see the Table objects for views, so add
it to the inputs
// now. isInsideView will tell if this view is embedded in another view.
ReadEntity viewInput = new ReadEntity(tab, parentInput,
!qb.isInsideView());
viewInput = PlanUtils.addInput(inputs, viewInput);
aliasToViewInfo.put(alias, new ObjectPair(fullViewName, viewInput));
viewAliasToInput.put(getAliasId(alias, qb), viewInput);
{noformat}
The key is getAliasId(alias, qb), the key is non-processed alias ID, for the
union all case, it looks like
null-subquery1:_u1-subquery1:viewsrc
But in method getParentViewInfo, alias ID is processed before get from the map.
The alias ID replace subquery1 to empty, the key
looks like following:
null:_u1:viewsrc
{noformat}
public static ReadEntity getParentViewInfo(String alias_id,
Map viewAliasToInput) {
String[] aliases = alias_id.split(":");
String currentAlias = null;
ReadEntity currentInput = null;
// Find the immediate parent possible.
// For eg: for a query like 'select * from V3', where V3 -> V2, V2 -> V1,
V1 -> T
// -> implies depends on.
// T's parent would be V1
for (int pos = 0; pos < aliases.length; pos++) {
currentAlias = currentAlias == null ? aliases[pos] : currentAlias + ":" +
aliases[pos];
currentAlias = currentAlias.replace(SemanticAnalyzer.SUBQUERY_TAG_1, "")
.replace(SemanticAnalyzer.SUBQUERY_TAG_2, "");
ReadEntity input = viewAliasToInput.get(currentAlias);
{noformat}
This inconsistence causes getParentViewInfo return null.
Attach the patch to fix the issue.
> Union All on view fail with no valid permission on underneath table
> ---
>
> Key: HIVE-13991
> URL: https://issues.apache.org/jira/browse/HIVE-13991
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>
> When sentry is enabled.
> create view V as select * from T;
> When the user has read permission on view V, but does not have read
> permission on table T,
> select * from V union all select * from V
> failed with:
> {noformat}
> 0: jdbc:hive2://> select * from s07view union all select * from
> s07view limit 1;
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privileges for this query:
> Server=server1->Db=default->Table=sample_07->action=select;
> (state=42000,code=4)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13913) LLAP: introduce backpressure to recordreader
[ https://issues.apache.org/jira/browse/HIVE-13913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323756#comment-15323756 ] Sergey Shelukhin commented on HIVE-13913: - Significantly modified patch. > LLAP: introduce backpressure to recordreader > > > Key: HIVE-13913 > URL: https://issues.apache.org/jira/browse/HIVE-13913 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13913.01.patch, HIVE-13913.02.patch, > HIVE-13913.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13913) LLAP: introduce backpressure to recordreader
[ https://issues.apache.org/jira/browse/HIVE-13913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13913: Attachment: HIVE-13913.02.patch > LLAP: introduce backpressure to recordreader > > > Key: HIVE-13913 > URL: https://issues.apache.org/jira/browse/HIVE-13913 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13913.01.patch, HIVE-13913.02.patch, > HIVE-13913.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13866) flatten callstack for directSQL errors
[ https://issues.apache.org/jira/browse/HIVE-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323735#comment-15323735 ] Ashutosh Chauhan commented on HIVE-13866: - +1 > flatten callstack for directSQL errors > -- > > Key: HIVE-13866 > URL: https://issues.apache.org/jira/browse/HIVE-13866 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13866.01.patch, HIVE-13866.patch > > > These errors look like final errors and confuse people. The callstack may be > useful if it's some datanucleus/db issue, but it needs to be flattened and > logged with a warning that this is not a final query error and that there's a > fallback -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13866) flatten callstack for directSQL errors
[ https://issues.apache.org/jira/browse/HIVE-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13866: Attachment: HIVE-13866.01.patch Updated > flatten callstack for directSQL errors > -- > > Key: HIVE-13866 > URL: https://issues.apache.org/jira/browse/HIVE-13866 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13866.01.patch, HIVE-13866.patch > > > These errors look like final errors and confuse people. The callstack may be > useful if it's some datanucleus/db issue, but it needs to be flattened and > logged with a warning that this is not a final query error and that there's a > fallback -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13913) LLAP: introduce backpressure to recordreader
[ https://issues.apache.org/jira/browse/HIVE-13913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13913: Attachment: (was: HIVE-13913.02.patch) > LLAP: introduce backpressure to recordreader > > > Key: HIVE-13913 > URL: https://issues.apache.org/jira/browse/HIVE-13913 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13913.01.patch, HIVE-13913.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323691#comment-15323691 ] Jason Dere commented on HIVE-13771: --- +1 > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13771.01.patch, HIVE-13771.01.wo.13731.patch, > HIVE-13771.patch > > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13913) LLAP: introduce backpressure to recordreader
[ https://issues.apache.org/jira/browse/HIVE-13913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13913: Attachment: HIVE-13913.02.patch > LLAP: introduce backpressure to recordreader > > > Key: HIVE-13913 > URL: https://issues.apache.org/jira/browse/HIVE-13913 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13913.01.patch, HIVE-13913.02.patch, > HIVE-13913.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13264) JDBC driver makes 2 Open Session Calls for every open session
[
https://issues.apache.org/jira/browse/HIVE-13264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323668#comment-15323668
]
Hive QA commented on HIVE-13264:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809094/HIVE-13264.9.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10223 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/66/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/66/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-66/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12809094 - PreCommit-HIVE-MASTER-Build
> JDBC driver makes 2 Open Session Calls for every open session
> -
>
> Key: HIVE-13264
> URL: https://issues.apache.org/jira/browse/HIVE-13264
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: NITHIN MAHESH
>Assignee: NITHIN MAHESH
> Labels: jdbc
> Attachments: HIVE-13264.1.patch, HIVE-13264.2.patch,
> HIVE-13264.3.patch, HIVE-13264.4.patch, HIVE-13264.5.patch,
> HIVE-13264.6.patch, HIVE-13264.6.patch, HIVE-13264.7.patch,
> HIVE-13264.8.patch, HIVE-13264.9.patch, HIVE-13264.patch
>
>
> When HTTP is used as the transport mode by the Hive JDBC driver, we noticed
> that there is an additional open/close session just to validate the
> connection.
>
> TCLIService.Iface client = new TCLIService.Client(new
> TBinaryProtocol(transport));
> TOpenSessionResp openResp = client.OpenSession(new TOpenSessionReq());
> if (openResp != null) {
> client.CloseSession(new
> TCloseSessionReq(openResp.getSessionHandle()));
> }
>
> The open session call is a costly one and should not be used to test
> transport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13989) Extended ACLs are not handled according to specification
[ https://issues.apache.org/jira/browse/HIVE-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Drome updated HIVE-13989: --- Affects Version/s: 2.0.0 Target Version/s: 2.0.0, 1.2.1 (was: 1.2.1) > Extended ACLs are not handled according to specification > > > Key: HIVE-13989 > URL: https://issues.apache.org/jira/browse/HIVE-13989 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 1.2.1, 2.0.0 >Reporter: Chris Drome >Assignee: Chris Drome > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13756) Map failure attempts to delete reducer _temporary directory on multi-query pig query
[ https://issues.apache.org/jira/browse/HIVE-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Drome updated HIVE-13756: --- Target Version/s: 2.0.0, 1.2.1 Fix Version/s: (was: 2.0.0) (was: 1.2.1) > Map failure attempts to delete reducer _temporary directory on multi-query > pig query > > > Key: HIVE-13756 > URL: https://issues.apache.org/jira/browse/HIVE-13756 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 1.2.1, 2.0.0 >Reporter: Chris Drome >Assignee: Chris Drome > Attachments: HIVE-13756-branch-1.patch, HIVE-13756.patch > > > A pig script, executed with multi-query enabled, that reads the source data > and writes it as-is into TABLE_A as well as performing a group-by operation > on the data which is written into TABLE_B can produce erroneous results if > any map fails. This results in a single MR job that writes the map output to > a scratch directory relative to TABLE_A and the reducer output to a scratch > directory relative to TABLE_B. > If one or more maps fail it will delete the attempt data relative to TABLE_A, > but it also deletes the _temporary directory relative to TABLE_B. This has > the unintended side-effect of preventing subsequent maps from committing > their data. This means that any maps which successfully completed before the > first map failure will have its data committed as expected, other maps not, > resulting in an incomplete result set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13754) Fix resource leak in HiveClientCache
[
https://issues.apache.org/jira/browse/HIVE-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Drome updated HIVE-13754:
---
Target Version/s: 2.0.0, 1.2.1
Fix Version/s: (was: 2.0.0)
(was: 1.2.1)
> Fix resource leak in HiveClientCache
>
>
> Key: HIVE-13754
> URL: https://issues.apache.org/jira/browse/HIVE-13754
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Chris Drome
>Assignee: Chris Drome
> Attachments: HIVE-13754-branch-1.patch, HIVE-13754.patch
>
>
> Found that the {{users}} reference count can go into negative values, which
> prevents {{tearDownIfUnused}} from closing the client connection when called.
> This leads to a build up of clients which have been evicted from the cache,
> are no longer in use, but have not been shutdown.
> GC will eventually call {{finalize}}, which forcibly closes the connection
> and cleans up the client, but I have seen as many as several hundred open
> client connections as a result.
> The main resource for this is caused by RetryingMetaStoreClient, which will
> call {{reconnect}} on acquire, which calls {{close}}. This will decrement
> {{users}} to -1 on the reconnect, then acquire will increase this to 0 while
> using it, and back to -1 when it releases it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13988) zero length file is being created for empty bucket in tez mode
[
https://issues.apache.org/jira/browse/HIVE-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13988:
---
Status: Patch Available (was: Open)
[~ashutoshc], could u take a look? Thanks.
> zero length file is being created for empty bucket in tez mode
> --
>
> Key: HIVE-13988
> URL: https://issues.apache.org/jira/browse/HIVE-13988
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13988.01.patch
>
>
> Even though bucket is empty, zero length file is being created in tez mode.
> steps to reproduce the issue:
> {noformat}
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> hive> drop table if exists emptybucket_orc;
> OK
> Time taken: 5.416 seconds
> hive> create table emptybucket_orc(age int) clustered by (age) sorted by
> (age) into 99 buckets stored as orc;
> OK
> Time taken: 0.493 seconds
> hive> insert into table emptybucket_orc select distinct(age) from
> studenttab10k limit 0;
> Query ID = hrt_qa_20160523231955_8b981be7-68c4-4416-8a48-5f8c7ff551c3
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1464045121842_0002)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 2 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 3 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 4 .. llap SUCCEEDED 99 9900
> 0 0
> --
> VERTICES: 04/04 [==>>] 100% ELAPSED TIME: 11.00 s
>
> --
> Loading data to table default.emptybucket_orc
> OK
> Time taken: 16.907 seconds
> hive> dfs -ls /apps/hive/warehouse/emptybucket_orc;
> Found 99 items
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/00_0
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/01_0
> ..
> {noformat}
> Expected behavior:
> In tez mode, zero length file shouldn't get created on hdfs if bucket is empty
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13988) zero length file is being created for empty bucket in tez mode
[
https://issues.apache.org/jira/browse/HIVE-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13988:
---
Attachment: HIVE-13988.01.patch
> zero length file is being created for empty bucket in tez mode
> --
>
> Key: HIVE-13988
> URL: https://issues.apache.org/jira/browse/HIVE-13988
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13988.01.patch
>
>
> Even though bucket is empty, zero length file is being created in tez mode.
> steps to reproduce the issue:
> {noformat}
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> hive> drop table if exists emptybucket_orc;
> OK
> Time taken: 5.416 seconds
> hive> create table emptybucket_orc(age int) clustered by (age) sorted by
> (age) into 99 buckets stored as orc;
> OK
> Time taken: 0.493 seconds
> hive> insert into table emptybucket_orc select distinct(age) from
> studenttab10k limit 0;
> Query ID = hrt_qa_20160523231955_8b981be7-68c4-4416-8a48-5f8c7ff551c3
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1464045121842_0002)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 2 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 3 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 4 .. llap SUCCEEDED 99 9900
> 0 0
> --
> VERTICES: 04/04 [==>>] 100% ELAPSED TIME: 11.00 s
>
> --
> Loading data to table default.emptybucket_orc
> OK
> Time taken: 16.907 seconds
> hive> dfs -ls /apps/hive/warehouse/emptybucket_orc;
> Found 99 items
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/00_0
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/01_0
> ..
> {noformat}
> Expected behavior:
> In tez mode, zero length file shouldn't get created on hdfs if bucket is empty
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323610#comment-15323610 ] Sravya Tirukkovalur commented on HIVE-13966: +1 to synchronous listener. Writing to notification log outside of transaction comes with potential correctness issues. As [~vaidyand] mentioned: 1. Operation fails, but notification entry is added: This can be avoided by checking the status of the operation 2. Operation succeeds but the notification entry is not written. This means data/metadata loss for downstream systems relying on NotificationLog. 3. Notification log also allows to get the linear event log when there are multiple HMSs in a active-active HMS HA setting. But, not having notification writing as part of transaction means, the commands can be out of order in notification log. > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Priority: Critical > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13380) Decimal should have lower precedence than double in type hierachy
[ https://issues.apache.org/jira/browse/HIVE-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323588#comment-15323588 ] Ashutosh Chauhan commented on HIVE-13380: - I think correct course of action here is to treat literal constants as big decimal instead of current choice of double. That will get us consistent behavior. I checked on MySQL and its doing the same thing. However that will likely require more thought and work. So, I think correct course of action is to revert this patch from branch-2.1 and then file a follow-up jira on master to infer non-integeral constant literals as big decimal. > Decimal should have lower precedence than double in type hierachy > - > > Key: HIVE-13380 > URL: https://issues.apache.org/jira/browse/HIVE-13380 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13380.2.patch, HIVE-13380.4.patch, > HIVE-13380.5.patch, HIVE-13380.patch, decimal_filter.q > > > Currently its other way round. Also, decimal should be lower than float. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-9995:
-
Description:
Consider TestWorker.minorWithOpenInMiddle()
since there is an open txnId=23, this doesn't have any meaningful minor
compaction work to do. The system still tries to compact a single delta file
for 21-22 id range, and effectively copies the file onto itself.
This is 1. inefficient and 2. can potentially affect a reader.
(from a real cluster)
Suppose we start with
{noformat}
drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
/user/hive/warehouse/t/base_016
-rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
/user/hive/warehouse/t/base_016/bucket_0
drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
/user/hive/warehouse/t/base_017
-rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
/user/hive/warehouse/t/base_017/bucket_0
drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
/user/hive/warehouse/t/delta_017_017_
-rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
/user/hive/warehouse/t/delta_017_017_/bucket_0
drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
/user/hive/warehouse/t/delta_018_018_
-rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
/user/hive/warehouse/t/delta_018_018_/bucket_0
{noformat}
then do _alter table T compact 'minor';_
then we end up with
{noformat}
drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
/user/hive/warehouse/t/base_017
-rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
/user/hive/warehouse/t/base_017/bucket_0
drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
/user/hive/warehouse/t/delta_018_018
-rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
/user/hive/warehouse/t/delta_018_018/bucket_0
drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
/user/hive/warehouse/t/delta_018_018_
-rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
/user/hive/warehouse/t/delta_018_018_/bucket_0
{noformat}
So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
was:
Consider TestWorker.minorWithOpenInMiddle()
since there is an open txnId=23, this doesn't have any meaningful minor
compaction work to do. The system still tries to compact a single delta file
for 21-22 id range, and effectively copies the file onto itself.
This is 1. inefficient and 2. can potentially affect a reader.
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1
[jira] [Updated] (HIVE-13984) Use multi-threaded approach to listing files for msck
[ https://issues.apache.org/jira/browse/HIVE-13984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13984: --- Attachment: HIVE-13984.01.patch > Use multi-threaded approach to listing files for msck > - > > Key: HIVE-13984 > URL: https://issues.apache.org/jira/browse/HIVE-13984 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13984.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13984) Use multi-threaded approach to listing files for msck
[ https://issues.apache.org/jira/browse/HIVE-13984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13984: --- Status: Patch Available (was: Open) > Use multi-threaded approach to listing files for msck > - > > Key: HIVE-13984 > URL: https://issues.apache.org/jira/browse/HIVE-13984 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13984.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13981) Operation.toSQLException eats full exception stack
[
https://issues.apache.org/jira/browse/HIVE-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323570#comment-15323570
]
Vaibhav Gumashta commented on HIVE-13981:
-
+1
> Operation.toSQLException eats full exception stack
> --
>
> Key: HIVE-13981
> URL: https://issues.apache.org/jira/browse/HIVE-13981
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-13981.1.patch
>
>
> Operation.toSQLException eats half of the exception stack and make debug
> hard. For example, we saw an exception:
> {code}
> org.apache.hive.service.cli.HiveSQL Exception : Error while compiling
> statement: FAILED : NullPointer Exception null
> at org.apache.hive.service.cli.operation.Operation.toSQL Exception
> (Operation.java:336)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:113)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:182)
> at org.apache.hive.service.cli.operation.Operation.run(Operation.java:278)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:421)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:408)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:276)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:505)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:562)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang. NullPointer Exception
> {code}
> The real stack causing the NPE is lost.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13981) Operation.toSQLException eats full exception stack
[
https://issues.apache.org/jira/browse/HIVE-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323569#comment-15323569
]
Daniel Dai commented on HIVE-13981:
---
Tested and now I can see full stack in hs2.log.
> Operation.toSQLException eats full exception stack
> --
>
> Key: HIVE-13981
> URL: https://issues.apache.org/jira/browse/HIVE-13981
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-13981.1.patch
>
>
> Operation.toSQLException eats half of the exception stack and make debug
> hard. For example, we saw an exception:
> {code}
> org.apache.hive.service.cli.HiveSQL Exception : Error while compiling
> statement: FAILED : NullPointer Exception null
> at org.apache.hive.service.cli.operation.Operation.toSQL Exception
> (Operation.java:336)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:113)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:182)
> at org.apache.hive.service.cli.operation.Operation.run(Operation.java:278)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:421)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:408)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:276)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:505)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:562)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang. NullPointer Exception
> {code}
> The real stack causing the NPE is lost.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-6311) Design a new logo?
[ https://issues.apache.org/jira/browse/HIVE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo resolved HIVE-6311. --- Resolution: Fixed I think we have waited long enough for submissions to the "new logo contest" > Design a new logo? > -- > > Key: HIVE-6311 > URL: https://issues.apache.org/jira/browse/HIVE-6311 > Project: Hive > Issue Type: Task >Reporter: Brock Noland > > I have heard some folks saying we should create a new logo so I am creating a > jira for their comment, -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13961) ACID: Major compaction fails to include the original bucket files if there's no delta directory
[ https://issues.apache.org/jira/browse/HIVE-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323559#comment-15323559 ] Wei Zheng commented on HIVE-13961: -- [~jcamachorodriguez] Go ahead for the RC w/o this patch. I need to resolve a test mismatch. After the RC, is it still possible to get this into 2.1.0? > ACID: Major compaction fails to include the original bucket files if there's > no delta directory > --- > > Key: HIVE-13961 > URL: https://issues.apache.org/jira/browse/HIVE-13961 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13961.1.patch, HIVE-13961.2.patch, > HIVE-13961.3.patch, HIVE-13961.4.patch, HIVE-13961.5.patch > > > The issue can be reproduced by steps below: > 1. Insert a row to Non-ACID table > 2. Convert Non-ACID to ACID table (i.e. set transactional=true table property) > 3. Perform Major compaction -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13961) ACID: Major compaction fails to include the original bucket files if there's no delta directory
[
https://issues.apache.org/jira/browse/HIVE-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323548#comment-15323548
]
Hive QA commented on HIVE-13961:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809073/HIVE-13961.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10225 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/65/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/65/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-65/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12809073 - PreCommit-HIVE-MASTER-Build
> ACID: Major compaction fails to include the original bucket files if there's
> no delta directory
> ---
>
> Key: HIVE-13961
> URL: https://issues.apache.org/jira/browse/HIVE-13961
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0, 2.1.0, 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>Priority: Blocker
> Attachments: HIVE-13961.1.patch, HIVE-13961.2.patch,
> HIVE-13961.3.patch, HIVE-13961.4.patch, HIVE-13961.5.patch
>
>
> The issue can be reproduced by steps below:
> 1. Insert a row to Non-ACID table
> 2. Convert Non-ACID to ACID table (i.e. set transactional=true table property)
> 3. Perform Major compaction
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13987) Clarify current error shown when HS2 is down
[
https://issues.apache.org/jira/browse/HIVE-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdullah Yousufi updated HIVE-13987:
Attachment: HIVE-13987.01.patch
> Clarify current error shown when HS2 is down
>
>
> Key: HIVE-13987
> URL: https://issues.apache.org/jira/browse/HIVE-13987
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13987.01.patch
>
>
> When HS2 is down and a query is run, the following error is shown in beeline:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}
> It may be more helpful to also indicate that the reason for this is that HS2
> is down, such as:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> HS2 may be unavailable, check server status
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13987) Clarify current error shown when HS2 is down
[
https://issues.apache.org/jira/browse/HIVE-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdullah Yousufi updated HIVE-13987:
Status: Patch Available (was: Open)
> Clarify current error shown when HS2 is down
>
>
> Key: HIVE-13987
> URL: https://issues.apache.org/jira/browse/HIVE-13987
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
>
> When HS2 is down and a query is run, the following error is shown in beeline:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}
> It may be more helpful to also indicate that the reason for this is that HS2
> is down, such as:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> HS2 may be unavailable, check server status
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13866) flatten callstack for directSQL errors
[ https://issues.apache.org/jira/browse/HIVE-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323539#comment-15323539 ] Ashutosh Chauhan commented on HIVE-13866: - I think we should have full stack trace at LOG.debug level so that devs can get to all frames while debugging. > flatten callstack for directSQL errors > -- > > Key: HIVE-13866 > URL: https://issues.apache.org/jira/browse/HIVE-13866 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13866.patch > > > These errors look like final errors and confuse people. The callstack may be > useful if it's some datanucleus/db issue, but it needs to be flattened and > logged with a warning that this is not a final query error and that there's a > fallback -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13913) LLAP: introduce backpressure to recordreader
[ https://issues.apache.org/jira/browse/HIVE-13913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323533#comment-15323533 ] Sergey Shelukhin commented on HIVE-13913: - Oh. nm, that is dumb > LLAP: introduce backpressure to recordreader > > > Key: HIVE-13913 > URL: https://issues.apache.org/jira/browse/HIVE-13913 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13913.01.patch, HIVE-13913.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13866) flatten callstack for directSQL errors
[
https://issues.apache.org/jira/browse/HIVE-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323531#comment-15323531
]
Sergey Shelukhin commented on HIVE-13866:
-
[~ashutoshc] before:
{noformat}
2016-06-09T15:33:44,595 WARN [main[]]: metastore.ObjectStore
(ObjectStore.java:handleDirectSqlError(2699)) - Direct SQL failed, falling back
to ORM
java.lang.RuntimeException: boom!
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getDatabase(MetaStoreDirectSql.java:292)
~[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$1.getSqlResult(ObjectStore.java:641)
~[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$1.getSqlResult(ObjectStore.java:638)
~[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2664)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.getDatabaseInternal(ObjectStore.java:638)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:622)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_45]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_45]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_45]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at com.sun.proxy.$Proxy26.getDatabase(Unknown Source) [?:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:586)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:608)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:398)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:78)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6396)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:238)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[?:1.8.0_45]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[?:1.8.0_45]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[?:1.8.0_45]
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
[?:1.8.0_45]
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1625)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:80)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3317)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3356)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3336)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3590)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)
[jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323525#comment-15323525
]
Brock Noland commented on HIVE-13884:
-
Can you test on a MySQL MS with a table with 100K partitions how much latency
this adds on average?
Otherwise it's reasonable.
> Disallow queries fetching more than a configured number of partitions in
> PartitionPruner
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13913) LLAP: introduce backpressure to recordreader
[ https://issues.apache.org/jira/browse/HIVE-13913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323524#comment-15323524 ] Sergey Shelukhin commented on HIVE-13913: - Somehow both consumers and producers are stuck on the queue. Marvelous... patch is not ready > LLAP: introduce backpressure to recordreader > > > Key: HIVE-13913 > URL: https://issues.apache.org/jira/browse/HIVE-13913 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13913.01.patch, HIVE-13913.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[
https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323511#comment-15323511
]
Nachiket Vaidya commented on HIVE-13966:
Another clean solution is to have synchronous notification listeners which run
notification operations in the same transaction.
For example:
{noformat}
hive.metastore.synchronous.event.listeners
XXX
{noformat}
[~sushanth] Do you have any inputs?
> DbNotificationListener: can loose DDL operation notifications
> -
>
> Key: HIVE-13966
> URL: https://issues.apache.org/jira/browse/HIVE-13966
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Nachiket Vaidya
>Priority: Critical
>
> The code for each API in HiveMetaStore.java is like this:
> 1. openTransaction()
> 2. -- operation--
> 3. commit() or rollback() based on result of the operation.
> 4. add entry to notification log (unconditionally)
> If the operation is failed (in step 2), we still add entry to notification
> log. Found this issue in testing.
> It is still ok as this is the case of false positive.
> If the operation is successful and adding to notification log failed, the
> user will get an MetaException. It will not rollback the operation, as it is
> already committed. We need to handle this case so that we will not have false
> negatives.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13961) ACID: Major compaction fails to include the original bucket files if there's no delta directory
[ https://issues.apache.org/jira/browse/HIVE-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323508#comment-15323508 ] Jesus Camacho Rodriguez commented on HIVE-13961: Sure, sounds good. Thanks > ACID: Major compaction fails to include the original bucket files if there's > no delta directory > --- > > Key: HIVE-13961 > URL: https://issues.apache.org/jira/browse/HIVE-13961 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13961.1.patch, HIVE-13961.2.patch, > HIVE-13961.3.patch, HIVE-13961.4.patch, HIVE-13961.5.patch > > > The issue can be reproduced by steps below: > 1. Insert a row to Non-ACID table > 2. Convert Non-ACID to ACID table (i.e. set transactional=true table property) > 3. Perform Major compaction -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13961) ACID: Major compaction fails to include the original bucket files if there's no delta directory
[ https://issues.apache.org/jira/browse/HIVE-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323499#comment-15323499 ] Wei Zheng commented on HIVE-13961: -- [~jcamachorodriguez] Can you wait for ~30 minutes? Right now it's being tested by Precommit (#65). If this run is clean I think I can get it committed quickly. If not clean, then we can skip this one for now. > ACID: Major compaction fails to include the original bucket files if there's > no delta directory > --- > > Key: HIVE-13961 > URL: https://issues.apache.org/jira/browse/HIVE-13961 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13961.1.patch, HIVE-13961.2.patch, > HIVE-13961.3.patch, HIVE-13961.4.patch, HIVE-13961.5.patch > > > The issue can be reproduced by steps below: > 1. Insert a row to Non-ACID table > 2. Convert Non-ACID to ACID table (i.e. set transactional=true table property) > 3. Perform Major compaction -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13961) ACID: Major compaction fails to include the original bucket files if there's no delta directory
[ https://issues.apache.org/jira/browse/HIVE-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323486#comment-15323486 ] Jesus Camacho Rodriguez commented on HIVE-13961: [~wzheng], is this ready to go in? It is the only blocker for the next 2.1.0 RC. Thanks > ACID: Major compaction fails to include the original bucket files if there's > no delta directory > --- > > Key: HIVE-13961 > URL: https://issues.apache.org/jira/browse/HIVE-13961 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13961.1.patch, HIVE-13961.2.patch, > HIVE-13961.3.patch, HIVE-13961.4.patch, HIVE-13961.5.patch > > > The issue can be reproduced by steps below: > 1. Insert a row to Non-ACID table > 2. Convert Non-ACID to ACID table (i.e. set transactional=true table property) > 3. Perform Major compaction -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
[ https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323485#comment-15323485 ] Matt McCline commented on HIVE-13432: - We might have a clue of what is causing this problem. We have one test case where "minor" compaction fails when hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat but succeeds when that environment variable is org.apache.hadoop.hive.ql.io.HiveInputFormat > ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7 > --- > > Key: HIVE-13432 > URL: https://issues.apache.org/jira/browse/HIVE-13432 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 > Environment: Hadoop 2.6.2+Hive 1.2.1 >Reporter: Qiuzhuang Lian >Assignee: Matt McCline > > After initiating HIVE ACID ORC table compaction, the CompactorMR job throws > exception: > Error: java.lang.ArrayIndexOutOfBoundsException: 7 > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969) > at > org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > As a result, we see hadoop exception stack, > 297 failed with state FAILED due to: Task failed > task_1458819387386_11297_m_08 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > 2016-04-06 11:30:57,891 INFO [dn209006-27]: mapreduce.Job > (Job.java:monitorAndPrintJob(1392)) - Counters: 14 > Job Counters > Failed map tasks=16 > Killed map tasks=7 > Launched map tasks=23 > Other local map tasks=13 > Data-local map tasks=6 > Rack-local map tasks=4 > Total time spent by all maps in occupied slots (ms)=412592 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=206296 > Total vcore-seconds taken by all map tasks=206296 > Total megabyte-seconds taken by all map tasks=422494208 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker > (Worker.java:run(176)) - Caught exception while trying to compact > lqz.my_orc_acid_table. Marking clean to avoid repeated failures, > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at > org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186) > at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162) > 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler > (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least > one row from completed_txn_components when marking compaction entry as clean! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13986) LLAP: kill Tez AM on token errors from plugin
[ https://issues.apache.org/jira/browse/HIVE-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323439#comment-15323439 ] Sergey Shelukhin commented on HIVE-13986: - no, it's ([Invalid token] or [remote-ex with invalid token in message]) and [llap-token in message] I'll move the method to after logging > LLAP: kill Tez AM on token errors from plugin > - > > Key: HIVE-13986 > URL: https://issues.apache.org/jira/browse/HIVE-13986 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13986.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13986) LLAP: kill Tez AM on token errors from plugin
[
https://issues.apache.org/jira/browse/HIVE-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323430#comment-15323430
]
Siddharth Seth commented on HIVE-13986:
---
{code}
if (((cause instanceof InvalidToken && cause.getMessage() != null)
|| (cause instanceof RemoteException && cause.getCause() ==
null
&& cause.getMessage() != null &&
cause.getMessage().contains("InvalidToken")))
&& cause.getMessage().contains(LLAP_TOKEN_NAME)) {
{code}
Is a closing bracket missed on the last line, and instead exists on the line
above. i.e should this be
{code}
&& cause.getMessage() != null && cause.getMessage().contains("InvalidToken"))
&& cause.getMessage().contains(LLAP_TOKEN_NAME)))
{code}
API usage looks good. If this makes sense, we can customize
ServicePluginErrorDefaults.OTHER_FATAL to an enum defined in Hive which is more
specific than OTHER_FATAL.
Can the method be called after logging from all the functions ?
> LLAP: kill Tez AM on token errors from plugin
> -
>
> Key: HIVE-13986
> URL: https://issues.apache.org/jira/browse/HIVE-13986
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13986.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13380) Decimal should have lower precedence than double in type hierachy
[ https://issues.apache.org/jira/browse/HIVE-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323412#comment-15323412 ] Sergey Shelukhin edited comment on HIVE-13380 at 6/9/16 9:36 PM: - This is why we should revert this patch :P Other changes may also be required. I don't have access to a newer SQL standard, we can double check (ahem), but I don't think it's something that would have changed. was (Author: sershe): This is why we should revert this patch :P > Decimal should have lower precedence than double in type hierachy > - > > Key: HIVE-13380 > URL: https://issues.apache.org/jira/browse/HIVE-13380 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13380.2.patch, HIVE-13380.4.patch, > HIVE-13380.5.patch, HIVE-13380.patch, decimal_filter.q > > > Currently its other way round. Also, decimal should be lower than float. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13380) Decimal should have lower precedence than double in type hierachy
[ https://issues.apache.org/jira/browse/HIVE-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323412#comment-15323412 ] Sergey Shelukhin commented on HIVE-13380: - This is why we should revert this patch :P > Decimal should have lower precedence than double in type hierachy > - > > Key: HIVE-13380 > URL: https://issues.apache.org/jira/browse/HIVE-13380 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13380.2.patch, HIVE-13380.4.patch, > HIVE-13380.5.patch, HIVE-13380.patch, decimal_filter.q > > > Currently its other way round. Also, decimal should be lower than float. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13380) Decimal should have lower precedence than double in type hierachy
[ https://issues.apache.org/jira/browse/HIVE-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323395#comment-15323395 ] Jason Dere commented on HIVE-13380: --- Unfortunately in Hive, the literal 0.06 is actually a double value. Try 0.06BD (which specifies decimal literal) to see the difference > Decimal should have lower precedence than double in type hierachy > - > > Key: HIVE-13380 > URL: https://issues.apache.org/jira/browse/HIVE-13380 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13380.2.patch, HIVE-13380.4.patch, > HIVE-13380.5.patch, HIVE-13380.patch, decimal_filter.q > > > Currently its other way round. Also, decimal should be lower than float. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13380) Decimal should have lower precedence than double in type hierachy
[ https://issues.apache.org/jira/browse/HIVE-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323381#comment-15323381 ] Sergey Shelukhin commented on HIVE-13380: - Yes it does. 0.06 and 0.01 are each exact , so the result should be exact. Then, column type is decimal, so there are no approximate types anywhere. > Decimal should have lower precedence than double in type hierachy > - > > Key: HIVE-13380 > URL: https://issues.apache.org/jira/browse/HIVE-13380 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13380.2.patch, HIVE-13380.4.patch, > HIVE-13380.5.patch, HIVE-13380.patch, decimal_filter.q > > > Currently its other way round. Also, decimal should be lower than float. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13833) Add an initial delay when starting the heartbeat
[ https://issues.apache.org/jira/browse/HIVE-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13833: - Attachment: HIVE-13833.2.patch patch 2 solves an issue where queryId is null when the heartbeat is started > Add an initial delay when starting the heartbeat > > > Key: HIVE-13833 > URL: https://issues.apache.org/jira/browse/HIVE-13833 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0, 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Minor > Attachments: HIVE-13833.1.patch, HIVE-13833.2.patch > > > Since the scheduling of heartbeat happens immediately after lock acquisition, > it's unnecessary to send heartbeat at the time when locks is acquired. Add an > initial delay to skip this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13986) LLAP: kill Tez AM on token errors from plugin
[ https://issues.apache.org/jira/browse/HIVE-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13986: Attachment: HIVE-13986.patch [~sseth] can you take a look wrt plugin API usage? Simple patch otherwise. > LLAP: kill Tez AM on token errors from plugin > - > > Key: HIVE-13986 > URL: https://issues.apache.org/jira/browse/HIVE-13986 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13986.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13986) LLAP: kill Tez AM on token errors from plugin
[ https://issues.apache.org/jira/browse/HIVE-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13986: Status: Patch Available (was: Open) > LLAP: kill Tez AM on token errors from plugin > - > > Key: HIVE-13986 > URL: https://issues.apache.org/jira/browse/HIVE-13986 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13986.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13986) LLAP: kill Tez AM on token errors from plugin
[ https://issues.apache.org/jira/browse/HIVE-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-13986: --- Assignee: Sergey Shelukhin > LLAP: kill Tez AM on token errors from plugin > - > > Key: HIVE-13986 > URL: https://issues.apache.org/jira/browse/HIVE-13986 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13380) Decimal should have lower precedence than double in type hierachy
[ https://issues.apache.org/jira/browse/HIVE-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323364#comment-15323364 ] Ashutosh Chauhan commented on HIVE-13380: - This doesn't describe this situation: bq. If the data type of *both* operands > Decimal should have lower precedence than double in type hierachy > - > > Key: HIVE-13380 > URL: https://issues.apache.org/jira/browse/HIVE-13380 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13380.2.patch, HIVE-13380.4.patch, > HIVE-13380.5.patch, HIVE-13380.patch, decimal_filter.q > > > Currently its other way round. Also, decimal should be lower than float. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in
[ https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abdullah Yousufi updated HIVE-13964: Attachment: HIVE-13964.02.patch > Add a parameter to beeline to allow a properties file to be passed in > - > > Key: HIVE-13964 > URL: https://issues.apache.org/jira/browse/HIVE-13964 > Project: Hive > Issue Type: New Feature > Components: Beeline >Affects Versions: 2.0.1 >Reporter: Abdullah Yousufi >Assignee: Abdullah Yousufi >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch > > > HIVE-6652 removed the ability to pass in a properties file as a beeline > parameter. It may be a useful feature to be able to pass the file in is a > parameter, such as --property-file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13380) Decimal should have lower precedence than double in type hierachy
[
https://issues.apache.org/jira/browse/HIVE-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323348#comment-15323348
]
Sergey Shelukhin edited comment on HIVE-13380 at 6/9/16 9:03 PM:
-
Per SQL92:
{noformat}
5.3
::=
[ [ ] ]
|
::= E
13)The data type of an is exact numeric.
The precision of an is the number of
s that it contains. The scale of an is the number of s to the right of the .
6.12
1) If the data type of both operands of a dyadic arithmetic opera-
tor is exact numeric, then the data type of the result is exact
numeric, with precision and scale determined as follows:
a) Let S1 and S2 be the scale of the first and second operands
respectively.
b) The precision of the result of addition and subtraction is
implementation-defined, and the scale is the maximum of S1
and S2.
c) The precision of the result of multiplication is implementation-
defined, and the scale is S1 + S2.
d) The precision and scale of the result of division is
implementation-defined.
{noformat}
Exact numeric is basically decimal in this case
was (Author: sershe):
Per SQL92:
{noformat}
5.3
::=
[ [ ] ]
|
::= E
13)The data type of an is exact numeric.
The precision of an is the number of
s that it contains. The scale of an is the number of s to the right of the .
6.12
1) If the data type of both operands of a dyadic arithmetic opera-
tor is exact numeric, then the data type of the result is exact
numeric, with precision and scale determined as follows:
a) Let S1 and S2 be the scale of the first and second operands
respectively.
b) The precision of the result of addition and subtraction is
implementation-defined, and the scale is the maximum of S1
and S2.
c) The precision of the result of multiplication is implementation-
defined, and the scale is S1 + S2.
d) The precision and scale of the result of division is
implementation-defined.
{noformat}
> Decimal should have lower precedence than double in type hierachy
> -
>
> Key: HIVE-13380
> URL: https://issues.apache.org/jira/browse/HIVE-13380
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-13380.2.patch, HIVE-13380.4.patch,
> HIVE-13380.5.patch, HIVE-13380.patch, decimal_filter.q
>
>
> Currently its other way round. Also, decimal should be lower than float.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13380) Decimal should have lower precedence than double in type hierachy
[
https://issues.apache.org/jira/browse/HIVE-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323348#comment-15323348
]
Sergey Shelukhin commented on HIVE-13380:
-
Per SQL92:
{noformat}
5.3
::=
[ [ ] ]
|
::= E
13)The data type of an is exact numeric.
The precision of an is the number of
s that it contains. The scale of an is the number of s to the right of the .
6.12
1) If the data type of both operands of a dyadic arithmetic opera-
tor is exact numeric, then the data type of the result is exact
numeric, with precision and scale determined as follows:
a) Let S1 and S2 be the scale of the first and second operands
respectively.
b) The precision of the result of addition and subtraction is
implementation-defined, and the scale is the maximum of S1
and S2.
c) The precision of the result of multiplication is implementation-
defined, and the scale is S1 + S2.
d) The precision and scale of the result of division is
implementation-defined.
{noformat}
> Decimal should have lower precedence than double in type hierachy
> -
>
> Key: HIVE-13380
> URL: https://issues.apache.org/jira/browse/HIVE-13380
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-13380.2.patch, HIVE-13380.4.patch,
> HIVE-13380.5.patch, HIVE-13380.patch, decimal_filter.q
>
>
> Currently its other way round. Also, decimal should be lower than float.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in
[ https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abdullah Yousufi updated HIVE-13964: Attachment: (was: HIVE-13964.02.patch) > Add a parameter to beeline to allow a properties file to be passed in > - > > Key: HIVE-13964 > URL: https://issues.apache.org/jira/browse/HIVE-13964 > Project: Hive > Issue Type: New Feature > Components: Beeline >Affects Versions: 2.0.1 >Reporter: Abdullah Yousufi >Assignee: Abdullah Yousufi >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-13964.01.patch > > > HIVE-6652 removed the ability to pass in a properties file as a beeline > parameter. It may be a useful feature to be able to pass the file in is a > parameter, such as --property-file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in
[ https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abdullah Yousufi updated HIVE-13964: Attachment: HIVE-13964.02.patch > Add a parameter to beeline to allow a properties file to be passed in > - > > Key: HIVE-13964 > URL: https://issues.apache.org/jira/browse/HIVE-13964 > Project: Hive > Issue Type: New Feature > Components: Beeline >Affects Versions: 2.0.1 >Reporter: Abdullah Yousufi >Assignee: Abdullah Yousufi >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch > > > HIVE-6652 removed the ability to pass in a properties file as a beeline > parameter. It may be a useful feature to be able to pass the file in is a > parameter, such as --property-file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13984) Use multi-threaded approach to listing files for msck
[ https://issues.apache.org/jira/browse/HIVE-13984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323337#comment-15323337 ] Pengcheng Xiong commented on HIVE-13984: cc'ing [~ashutoshc] > Use multi-threaded approach to listing files for msck > - > > Key: HIVE-13984 > URL: https://issues.apache.org/jira/browse/HIVE-13984 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12656) Turn hive.compute.query.using.stats on by default
[
https://issues.apache.org/jira/browse/HIVE-12656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323315#comment-15323315
]
Hive QA commented on HIVE-12656:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809061/HIVE-12656.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 108 failed/errored test(s), 10223 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_2_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_udf_udaf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_udf_udaf_stats_opt
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_udf_udaf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fileformat_mix
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_boolexpr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_decode_name
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_special_char
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_varchar1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_plan_json
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_constant_where
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rename_partition_location
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_unquote_and
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_unquote_not
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_unquote_or
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_aggregator_error_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_noscan_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_publisher_error_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
[jira] [Commented] (HIVE-13380) Decimal should have lower precedence than double in type hierachy
[ https://issues.apache.org/jira/browse/HIVE-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323288#comment-15323288 ] Ashutosh Chauhan commented on HIVE-13380: - hmm.. its arguable. As I noted in my previous comment if one of the sides in comparison is inexact there is no way to perform exact arithmetic operation. From their documentation both SQL Server & Oracle seems to treat decimal lower in hierarchy which motivated this change. Plus, since HiveDecimal is based on BigDecimal it is dog slow, so doing operation in Double would also be more performant. We can revert the patch but then we won't follow SQL Server & Oracle type's hierarchy which likely means we won't follow ansi standards. > Decimal should have lower precedence than double in type hierachy > - > > Key: HIVE-13380 > URL: https://issues.apache.org/jira/browse/HIVE-13380 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13380.2.patch, HIVE-13380.4.patch, > HIVE-13380.5.patch, HIVE-13380.patch, decimal_filter.q > > > Currently its other way round. Also, decimal should be lower than float. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13968) CombineHiveInputFormat does not honor InputFormat that implements AvoidSplitCombination
[ https://issues.apache.org/jira/browse/HIVE-13968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323255#comment-15323255 ] Prasanna Rajaperumal edited comment on HIVE-13968 at 6/9/16 8:15 PM: - Added the cleanup of tmp files created. [~lirui]. was (Author: prasanna@gmail.com): Added the cleanup of tmp files created. > CombineHiveInputFormat does not honor InputFormat that implements > AvoidSplitCombination > --- > > Key: HIVE-13968 > URL: https://issues.apache.org/jira/browse/HIVE-13968 > Project: Hive > Issue Type: Bug >Reporter: Prasanna Rajaperumal >Assignee: Prasanna Rajaperumal > Attachments: HIVE-13968.1.patch, HIVE-13968.2.patch, > HIVE-13968.3.patch > > > If I have 100 path[] , the nonCombinablePaths will have only the paths > paths[0-9] and the rest of the paths will be in combinablePaths, even if the > inputformat returns false for AvoidSplitCombination.shouldSkipCombine() for > all the paths. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13968) CombineHiveInputFormat does not honor InputFormat that implements AvoidSplitCombination
[ https://issues.apache.org/jira/browse/HIVE-13968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Rajaperumal updated HIVE-13968: Attachment: HIVE-13968.3.patch Added the cleanup of tmp files created. > CombineHiveInputFormat does not honor InputFormat that implements > AvoidSplitCombination > --- > > Key: HIVE-13968 > URL: https://issues.apache.org/jira/browse/HIVE-13968 > Project: Hive > Issue Type: Bug >Reporter: Prasanna Rajaperumal >Assignee: Prasanna Rajaperumal > Attachments: HIVE-13968.1.patch, HIVE-13968.2.patch, > HIVE-13968.3.patch > > > If I have 100 path[] , the nonCombinablePaths will have only the paths > paths[0-9] and the rest of the paths will be in combinablePaths, even if the > inputformat returns false for AvoidSplitCombination.shouldSkipCombine() for > all the paths. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13771: Attachment: HIVE-13771.01.wo.13731.patch HIVE-13771.01.patch The patches including the previous patch. [~sseth] [~jdere] this is a tiny patch, on top of the previous one... pretty much ready for review :) > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13771.01.patch, HIVE-13771.01.wo.13731.patch, > HIVE-13771.patch > > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13771: Attachment: (was: HIVE-13771.01.wo.13731.patch) > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13771.patch > > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13771: Attachment: (was: HIVE-13771.01.patch) > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13771.patch > > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13771: Attachment: HIVE-13771.01.patch HIVE-13771.01.wo.13731.patch > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13771.01.patch, HIVE-13771.01.wo.13731.patch, > HIVE-13771.patch > > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13731) LLAP: return LLAP token with the splits
[ https://issues.apache.org/jira/browse/HIVE-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323142#comment-15323142 ] Sergey Shelukhin commented on HIVE-13731: - [~sseth] [~jdere] can you take a look? relatively small patch :) > LLAP: return LLAP token with the splits > --- > > Key: HIVE-13731 > URL: https://issues.apache.org/jira/browse/HIVE-13731 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13731.01.patch, HIVE-13731.01.wo.13675-13443.patch, > HIVE-13731.02.patch, HIVE-13731.patch, HIVE-13731.wo.13444-13675-13443.patch > > > Need to return the token with the splits, then take it in LLAPIF and make > sure it's used when talking to LLAP -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13731) LLAP: return LLAP token with the splits
[ https://issues.apache.org/jira/browse/HIVE-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13731: Attachment: HIVE-13731.02.patch > LLAP: return LLAP token with the splits > --- > > Key: HIVE-13731 > URL: https://issues.apache.org/jira/browse/HIVE-13731 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13731.01.patch, HIVE-13731.01.wo.13675-13443.patch, > HIVE-13731.02.patch, HIVE-13731.patch, HIVE-13731.wo.13444-13675-13443.patch > > > Need to return the token with the splits, then take it in LLAPIF and make > sure it's used when talking to LLAP -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13675) LLAP: add HMAC signatures to LLAPIF splits
[ https://issues.apache.org/jira/browse/HIVE-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13675: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master > LLAP: add HMAC signatures to LLAPIF splits > -- > > Key: HIVE-13675 > URL: https://issues.apache.org/jira/browse/HIVE-13675 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.2.0 > > Attachments: HIVE-13675.01.patch, HIVE-13675.02.patch, > HIVE-13675.03.patch, HIVE-13675.04.patch, HIVE-13675.05.patch, > HIVE-13675.06.patch, HIVE-13675.07.patch, HIVE-13675.08.patch, > HIVE-13675.09.patch, HIVE-13675.10.patch, HIVE-13675.WIP.patch, > HIVE-13675.wo.13444.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13443) LLAP: signing for the second state of submit (the event)
[ https://issues.apache.org/jira/browse/HIVE-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13443: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master > LLAP: signing for the second state of submit (the event) > > > Key: HIVE-13443 > URL: https://issues.apache.org/jira/browse/HIVE-13443 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.2.0 > > Attachments: HIVE-13443.01.patch, HIVE-13443.02.patch, > HIVE-13443.02.wo.13675.nogen.patch, HIVE-13443.03.patch, HIVE-13443.03.patch, > HIVE-13443.03.wo.13675.nogen.patch, HIVE-13443.04.patch, HIVE-13443.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13567) Auto-gather column stats - phase 2
[
https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323083#comment-15323083
]
Hive QA commented on HIVE-13567:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809062/HIVE-13567.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 449 failed/errored test(s), 10224 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_numbuckets_partitioned_table2_h23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_numbuckets_partitioned_table_h23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_add_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_serde2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_table_null_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19_inclause
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binary_output_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_case_sensitivity
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_auto_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_gby2_map_multi_distinct
[jira] [Comment Edited] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in
[
https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323067#comment-15323067
]
Abdullah Yousufi edited comment on HIVE-13964 at 6/9/16 6:42 PM:
-
Thanks for the review, [~spena].
1. Fixed:
{code}
if (propertyFile != null) {
dispatch("!properties " + propertyFile);
}
{code}
2. That's strange, because I just tried that case and this was my output:
{code}
$ ./beeline --property-file
Missing argument for option: property-file
Usage: java org.apache.hive.cli.beeline.BeeLine
-uthe JDBC URL to connect to
...
{code}
Could you try another parameter without any arguments, such as --hiveconf and
see if it prints out the "Missing argument..." error for that?
3. Added a fix for this to exit, but can undo this if necessary.
Let me know about points 2 and 3, and then I can upload another patch.
was (Author: ayousufi):
Thanks for the review, Sergio.
1. Fixed:
{code}
if (propertyFile != null) {
dispatch("!properties " + propertyFile);
}
{code}
2. That's strange, because I just tried that case and this was my output:
{code}
$ ./beeline --property-file
Missing argument for option: property-file
Usage: java org.apache.hive.cli.beeline.BeeLine
-uthe JDBC URL to connect to
...
{code}
Could you try another parameter without any arguments, such as --hiveconf and
see if it prints out the "Missing argument..." error for that?
3. Added a fix for this to exit, but can undo this if necessary.
Let me know about points 2 and 3, and then I can upload another patch.
> Add a parameter to beeline to allow a properties file to be passed in
> -
>
> Key: HIVE-13964
> URL: https://issues.apache.org/jira/browse/HIVE-13964
> Project: Hive
> Issue Type: New Feature
> Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13964.01.patch
>
>
> HIVE-6652 removed the ability to pass in a properties file as a beeline
> parameter. It may be a useful feature to be able to pass the file in is a
> parameter, such as --property-file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in
[
https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323067#comment-15323067
]
Abdullah Yousufi commented on HIVE-13964:
-
Thanks for the review, Sergio.
1. Fixed:
{code}
if (propertyFile != null) {
dispatch("!properties " + propertyFile);
}
{code}
2. That's strange, because I just tried that case and this was my output:
{code}
$ ./beeline --property-file
Missing argument for option: property-file
Usage: java org.apache.hive.cli.beeline.BeeLine
-uthe JDBC URL to connect to
...
{code}
Could you try another parameter without any arguments, such as --hiveconf and
see if it prints out the "Missing argument..." error for that?
3. Added a fix for this to exit, but can undo this if necessary.
Let me know about points 2 and 3, and then I can upload another patch.
> Add a parameter to beeline to allow a properties file to be passed in
> -
>
> Key: HIVE-13964
> URL: https://issues.apache.org/jira/browse/HIVE-13964
> Project: Hive
> Issue Type: New Feature
> Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13964.01.patch
>
>
> HIVE-6652 removed the ability to pass in a properties file as a beeline
> parameter. It may be a useful feature to be able to pass the file in is a
> parameter, such as --property-file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13563: --- Target Version/s: 1.3.0, 2.1.0 (was: 1.3.0, 2.1.0, 2.2.0) > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13563.1.patch, HIVE-13563.2.patch, > HIVE-13563.3.patch, HIVE-13563.4.patch, HIVE-13563.branch-1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13563: --- Fix Version/s: (was: 2.2.0) > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13563.1.patch, HIVE-13563.2.patch, > HIVE-13563.3.patch, HIVE-13563.4.patch, HIVE-13563.branch-1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13958) hive.strict.checks.type.safety should apply to decimals, as well as IN... and BETWEEN... ops
[ https://issues.apache.org/jira/browse/HIVE-13958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13958: Assignee: Takuma Wakamori > hive.strict.checks.type.safety should apply to decimals, as well as IN... and > BETWEEN... ops > > > Key: HIVE-13958 > URL: https://issues.apache.org/jira/browse/HIVE-13958 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Takuma Wakamori > > String to decimal auto-casts should be prohibited for compares -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13981) Operation.toSQLException eats full exception stack
[
https://issues.apache.org/jira/browse/HIVE-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-13981:
--
Attachment: HIVE-13981.1.patch
Attach initial patch. Will test it shortly.
> Operation.toSQLException eats full exception stack
> --
>
> Key: HIVE-13981
> URL: https://issues.apache.org/jira/browse/HIVE-13981
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-13981.1.patch
>
>
> Operation.toSQLException eats half of the exception stack and make debug
> hard. For example, we saw an exception:
> {code}
> org.apache.hive.service.cli.HiveSQL Exception : Error while compiling
> statement: FAILED : NullPointer Exception null
> at org.apache.hive.service.cli.operation.Operation.toSQL Exception
> (Operation.java:336)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:113)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:182)
> at org.apache.hive.service.cli.operation.Operation.run(Operation.java:278)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:421)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:408)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:276)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:505)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:562)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang. NullPointer Exception
> {code}
> The real stack causing the NPE is lost.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13958) hive.strict.checks.type.safety should apply to decimals, as well as IN... and BETWEEN... ops
[ https://issues.apache.org/jira/browse/HIVE-13958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323053#comment-15323053 ] Sergey Shelukhin commented on HIVE-13958: - Done. Thanks for looking into this! > hive.strict.checks.type.safety should apply to decimals, as well as IN... and > BETWEEN... ops > > > Key: HIVE-13958 > URL: https://issues.apache.org/jira/browse/HIVE-13958 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Takuma Wakamori > > String to decimal auto-casts should be prohibited for compares -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13965) Empty resultset run into Exception when using Thrift Binary Serde
[
https://issues.apache.org/jira/browse/HIVE-13965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-13965:
Assignee: Ziyang Zhao
> Empty resultset run into Exception when using Thrift Binary Serde
> -
>
> Key: HIVE-13965
> URL: https://issues.apache.org/jira/browse/HIVE-13965
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
>
> This error can be reproduced by enabling thrift binary serde, using beeline
> connect to hiveserver2 and executing the following commands:
> >create table test3(num1 int);
> >create table test4(num1 int);
> >insert into test3 values(1);
> >insert into test4 values(2);
> >select * from test3 join test4 on test3.num1=test4.num1;
> The result should be empty, but it gives an exception:
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: Hive Runtime Error while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:206)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1029)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:641)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:195)
> ... 8 more
> This error is caused in FileSinkOperator.java.
> If the resultset is empty, function process() will not be called, so variable
> "fpaths" will not be set. When run into CloseOp(),
> if (conf.isHiveServerQuery() && HiveConf.getBoolVar(hconf,
> HiveConf.ConfVars.HIVE_SERVER2_THRIFT_RESULTSET_SERIALIZE_IN_TASKS)
> &&
>
> serializer.getClass().getName().equalsIgnoreCase(ThriftJDBCBinarySerDe.class.getName()))
> {
> try {
>recordValue = serializer.serialize(null, inputObjInspectors[0]);
>rowOutWriters = fpaths.outWriters;
>rowOutWriters[0].write(recordValue);
> } catch (SerDeException | IOException e) {
>throw new HiveException(e);
> }
> }
> Here fpaths is null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13982) Extension to limit push down through order by & group by
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13982:
---
Status: Patch Available (was: In Progress)
> Extension to limit push down through order by & group by
>
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.patch
>
>
> Queries which follow the format are not optimal with map-side aggregation,
> because the Map 1 does not have TopN in the reduce sink.
> These queries shuffle 100% of the aggregate in cases where the reduce de-dup
> does not kick in.
> As input data grows, it falls off a cliff of performance after 4 reducers.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13982) Extension to limit push down through order by & group by
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13982:
---
Attachment: HIVE-13982.patch
> Extension to limit push down through order by & group by
>
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.patch
>
>
> Queries which follow the format are not optimal with map-side aggregation,
> because the Map 1 does not have TopN in the reduce sink.
> These queries shuffle 100% of the aggregate in cases where the reduce de-dup
> does not kick in.
> As input data grows, it falls off a cliff of performance after 4 reducers.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13982) Extension to limit push down through order by & group by
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13982 started by Jesus Camacho Rodriguez.
--
> Extension to limit push down through order by & group by
>
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.patch
>
>
> Queries which follow the format are not optimal with map-side aggregation,
> because the Map 1 does not have TopN in the reduce sink.
> These queries shuffle 100% of the aggregate in cases where the reduce de-dup
> does not kick in.
> As input data grows, it falls off a cliff of performance after 4 reducers.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13563: - Resolution: Fixed Fix Version/s: 2.2.0 2.1.0 1.3.0 Status: Resolved (was: Patch Available) Committed to master, branch-2.1 and branch-1. Thanks Prasanth and Owen for the review. > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Fix For: 1.3.0, 2.1.0, 2.2.0 > > Attachments: HIVE-13563.1.patch, HIVE-13563.2.patch, > HIVE-13563.3.patch, HIVE-13563.4.patch, HIVE-13563.branch-1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13563: - Attachment: HIVE-13563.branch-1.patch > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch, HIVE-13563.2.patch, > HIVE-13563.3.patch, HIVE-13563.4.patch, HIVE-13563.branch-1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
