[
https://issues.apache.org/jira/browse/HIVE-18164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270709#comment-16270709
]
Dmitro-Vasilenko edited comment on HIVE-18164 at 11/29/17 1:49 PM:
-------------------------------------------------------------------
Also problem resolved after set
hive.tez.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat; <<< works
with TEZ
was (Author: dmitro-vasilenko):
Alse problem resolve after set
hive.tez.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat; <<< works
with TEZ
> Hive2 select with group by error if transactional = true
> ---------------------------------------------------------
>
> Key: HIVE-18164
> URL: https://issues.apache.org/jira/browse/HIVE-18164
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Transactions
> Affects Versions: 2.3.0
> Environment: Hortonworks HDP-2.6.3.0
> Reporter: Dmitro-Vasilenko
> Priority: Critical
>
> Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235)
> Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235)
> 0: jdbc:hive2://serv01:2181,ks-> select sum(destination),messagetype from
> t1.cdr where hday='2017-09-14' group by messagetype;
> INFO : Session is already open
> INFO : Dag name: select sum(destination),messag...messagetype(Stage-1)
> ERROR : Status: Failed
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1511771679762_0301_2_00, diagnostics=[Vertex
> vertex_1511771679762_0301_2_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: cdr initializer failed,
> vertex=vertex_1511771679762_0301_2_00 [Map 1], java.lang.RuntimeException:
> serious problem
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1277)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1304)
> at
> org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat.getSplits(BucketizedHiveInputFormat.java:141)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:448)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:300)
> at
> org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:266)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IllegalArgumentException: delta_16881612_29766798 does not start
> with base_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1254)
> ... 16 more
> Caused by: java.lang.IllegalArgumentException: delta_16881612_29766798 does
> not start with base_
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.parseBase(AcidUtils.java:190)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.parseBaseBucketFilename(AcidUtils.java:221)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.callInternal(OrcInputFormat.java:804)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.access$600(OrcInputFormat.java:747)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:772)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:769)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:769)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:747)
> ... 4 more
> ]
>
> Error occur if delta_* present : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>
> [serv01]$ hdfs dfs -ls /warehouse/t1/cdr/hday=2017-09-14
> Found 15 items
> drwxrwxrwx - hive hdfs 0 2017-09-16 11:29
> /warehouse/t1/cdr/hday=2017-09-14/base_16881497
> drwxrwxrwx - hive hdfs 0 2017-10-21 18:42
> /warehouse/t1/cdr/hday=2017-09-14/delta_16881612_29766798
> drwxr-xr-x - hive hdfs 0 2017-10-22 17:48
> /warehouse/t1/cdr/hday=2017-09-14/delta_30628231_30628231_0000
> drwxr-xr-x - hive hdfs 0 2017-10-26 18:06
> /warehouse/t1/cdr/hday=2017-09-14/delta_33418590_33418590_0000
> drwxr-xr-x - hive hdfs 0 2017-10-27 16:23
> /warehouse/t1/cdr/hday=2017-09-14/delta_33540229_33540229_0000
> drwxr-xr-x - hive hdfs 0 2017-10-27 16:33
> /warehouse/t1/cdr/hday=2017-09-14/delta_33541305_33541305_0000
> drwxr-xr-x - hive hdfs 0 2017-10-31 12:40
> /warehouse/t1/cdr/hday=2017-09-14/delta_34016509_34016509_0000
> drwxr-xr-x - hive hdfs 0 2017-10-31 13:30
> /warehouse/t1/cdr/hday=2017-09-14/delta_34025608_34025608_0000
> drwxr-xr-x - hive hdfs 0 2017-10-31 14:19
> /warehouse/t1/cdr/hday=2017-09-14/delta_34033668_34033668_0000
> drwxr-xr-x - hive hdfs 0 2017-11-01 21:38
> /warehouse/t1/cdr/hday=2017-09-14/delta_34219785_34219785_0000
> drwxr-xr-x - hive hdfs 0 2017-11-02 11:20
> /warehouse/t1/cdr/hday=2017-09-14/delta_34292833_34292833_0000
> drwxr-xr-x - hive hdfs 0 2017-11-10 09:52
> /warehouse/t1/cdr/hday=2017-09-14/delta_35449030_35449030_0000
> drwxr-xr-x - hive hdfs 0 2017-11-10 13:07
> /warehouse/t1/cdr/hday=2017-09-14/delta_35472185_35472185_0000
> drwxr-xr-x - hive hdfs 0 2017-11-13 19:07
> /warehouse/t1/cdr/hday=2017-09-14/delta_35944544_35944544_0000
> drwxr-xr-x - hive hdfs 0 2017-11-21 12:37
> /warehouse/t1/cdr/hday=2017-09-14/delta_36820930_36820930_0000
>
> Workaround:
>
> ALTER TABLE .. SET TBLPROPERTIES
> ('compactorthreshold.hive.compactor.delta.num.threshold'='1') - and wait
> done compactors working
>
> OR
>
> set hive.execution.engine=mr;
>
> Question :
>
> exist any other workaround for run select with TEZ ?
>
>
>
>
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
