Dmitro-Vasilenko created HIVE-18164:
---------------------------------------
Summary: Hive2 select with group by error if transactional = true
Key: HIVE-18164
URL: https://issues.apache.org/jira/browse/HIVE-18164
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 2.3.0
Environment: Hortonworks HDP-2.6.3.0
Reporter: Dmitro-Vasilenko
Priority: Critical
Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235)
Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235)
0: jdbc:hive2://serv01:2181,ks-> select sum(destination),messagetype from
t1.cdr where hday='2017-09-14' group by messagetype;
INFO : Session is already open
INFO : Dag name: select sum(destination),messag...messagetype(Stage-1)
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1,
vertexId=vertex_1511771679762_0301_2_00, diagnostics=[Vertex
vertex_1511771679762_0301_2_00 [Map 1] killed/failed due
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: cdr initializer failed,
vertex=vertex_1511771679762_0301_2_00 [Map 1], java.lang.RuntimeException:
serious problem
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1277)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1304)
at
org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat.getSplits(BucketizedHiveInputFormat.java:141)
at
org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:448)
at
org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:300)
at
org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:266)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: delta_16881612_29766798 does not start with
base_
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1254)
... 16 more
Caused by: java.lang.IllegalArgumentException: delta_16881612_29766798 does not
start with base_
at org.apache.hadoop.hive.ql.io.AcidUtils.parseBase(AcidUtils.java:190)
at
org.apache.hadoop.hive.ql.io.AcidUtils.parseBaseBucketFilename(AcidUtils.java:221)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.callInternal(OrcInputFormat.java:804)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.access$600(OrcInputFormat.java:747)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:772)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:769)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:769)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:747)
... 4 more
]
Error occur if delta_* present : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
[serv01]$ hdfs dfs -ls /warehouse/t1/cdr/hday=2017-09-14
Found 15 items
drwxrwxrwx - hive hdfs 0 2017-09-16 11:29
/warehouse/t1/cdr/hday=2017-09-14/base_16881497
drwxrwxrwx - hive hdfs 0 2017-10-21 18:42
/warehouse/t1/cdr/hday=2017-09-14/delta_16881612_29766798
drwxr-xr-x - hive hdfs 0 2017-10-22 17:48
/warehouse/t1/cdr/hday=2017-09-14/delta_30628231_30628231_0000
drwxr-xr-x - hive hdfs 0 2017-10-26 18:06
/warehouse/t1/cdr/hday=2017-09-14/delta_33418590_33418590_0000
drwxr-xr-x - hive hdfs 0 2017-10-27 16:23
/warehouse/t1/cdr/hday=2017-09-14/delta_33540229_33540229_0000
drwxr-xr-x - hive hdfs 0 2017-10-27 16:33
/warehouse/t1/cdr/hday=2017-09-14/delta_33541305_33541305_0000
drwxr-xr-x - hive hdfs 0 2017-10-31 12:40
/warehouse/t1/cdr/hday=2017-09-14/delta_34016509_34016509_0000
drwxr-xr-x - hive hdfs 0 2017-10-31 13:30
/warehouse/t1/cdr/hday=2017-09-14/delta_34025608_34025608_0000
drwxr-xr-x - hive hdfs 0 2017-10-31 14:19
/warehouse/t1/cdr/hday=2017-09-14/delta_34033668_34033668_0000
drwxr-xr-x - hive hdfs 0 2017-11-01 21:38
/warehouse/t1/cdr/hday=2017-09-14/delta_34219785_34219785_0000
drwxr-xr-x - hive hdfs 0 2017-11-02 11:20
/warehouse/t1/cdr/hday=2017-09-14/delta_34292833_34292833_0000
drwxr-xr-x - hive hdfs 0 2017-11-10 09:52
/warehouse/t1/cdr/hday=2017-09-14/delta_35449030_35449030_0000
drwxr-xr-x - hive hdfs 0 2017-11-10 13:07
/warehouse/t1/cdr/hday=2017-09-14/delta_35472185_35472185_0000
drwxr-xr-x - hive hdfs 0 2017-11-13 19:07
/warehouse/t1/cdr/hday=2017-09-14/delta_35944544_35944544_0000
drwxr-xr-x - hive hdfs 0 2017-11-21 12:37
/warehouse/t1/cdr/hday=2017-09-14/delta_36820930_36820930_0000
Workaround:
ALTER TABLE .. SET TBLPROPERTIES
('compactorthreshold.hive.compactor.delta.num.threshold'='1') - and wait done
compactors working
set hive.execution.engine=mr;
Question :
exist any other workaround for run select with TEZ ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
