[jira] [Created] (HIVE-21111) ConditionalTask cannot be cast to MapRedTask
zhuwei created HIVE-2:
-
Summary: ConditionalTask cannot be cast to MapRedTask
Key: HIVE-2
URL: https://issues.apache.org/jira/browse/HIVE-2
Project: Hive
Issue Type: Bug
Reporter: zhuwei
Assignee: zhuwei
We met error like this in our product environment:
java.lang.ClassCastException: org.apache.hadoop.hive.ql.exec.ConditionalTask
cannot be cast to org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at
org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:173)
There is a bug in function
org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch:
if (tsk.isMapRedTask()) {
Task newTask = this.processCurrentTask((MapRedTask)
tsk,
((ConditionalTask) currTask), physicalContext.getContext());
walkerCtx.addToDispatchList(newTask);
}
In the above code, when tsk is instance of ConditionalTask, tsk.isMapRedTask()
still can be true, but it cannot be cast to MapRedTask.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20725) Simultaneous dynamic inserts can result in partition files lost
zhuwei created HIVE-20725: - Summary: Simultaneous dynamic inserts can result in partition files lost Key: HIVE-20725 URL: https://issues.apache.org/jira/browse/HIVE-20725 Project: Hive Issue Type: Bug Reporter: zhuwei Assignee: zhuwei If two users attempt a dynamic insert into the same new partition at the same time, a possible race condition exists which result in error state. In that case the partition info has been inserted to metastore but data files been removed. The current logic in function "add_partition_core" in class HiveMetaStore.HMSHandler is like this : # check if partition already exists # create the partition files directory if not exists # try to add partition # if add partition failed and it created the directory in step 2, delete that directory Assume that two users are inserting the same partition at the same time, there are two threads operating their requests, say thread A and thread B. If 1~4 steps of thread B are all done between step 2 and step 3 of thread A. The sequence like this : A1 A2 B1 B2 B3 B4 A3 A4. The partition files written by B will be removed by A. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20497) ParseException, failed to recognize quoted identifier when re-parsing the re-written query
zhuwei created HIVE-20497: - Summary: ParseException, failed to recognize quoted identifier when re-parsing the re-written query Key: HIVE-20497 URL: https://issues.apache.org/jira/browse/HIVE-20497 Project: Hive Issue Type: Bug Components: Parser Environment: hive 2.1.1 Reporter: zhuwei Assignee: zhuwei select `user` from team; If we have a table `team`, and one of its column has been masked out with `` with column level authorization. The above query will fail with error "SemanticException org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 Failed to recognize predicate 'user'. Failed rule: 'identifier' in expression specification" The root cause is that after re-written the ast, the back quote has been lost. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19450) OOM due to map join and backup task not invoked
zhuwei created HIVE-19450: - Summary: OOM due to map join and backup task not invoked Key: HIVE-19450 URL: https://issues.apache.org/jira/browse/HIVE-19450 Project: Hive Issue Type: Bug Reporter: zhuwei Assignee: zhuwei Map join task may cause OOM due to orc compression , in most cases , a backup task will be invoked. However , if the size of hash table is close to memory limit, the task which load the hash table will NOT fail . OOM will happen in next task witch do local join. The load task has a backup but next task not. So in this case , the whole query will fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19287) parse error with semicolon in comment in file
zhuwei created HIVE-19287: - Summary: parse error with semicolon in comment in file Key: HIVE-19287 URL: https://issues.apache.org/jira/browse/HIVE-19287 Project: Hive Issue Type: Bug Environment: hive 2.2.1 Reporter: zhuwei Assignee: zhuwei It will get error when hive query written in file look like this: select col --this is; an example from db.table limit 1; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19202) CBO failed due to NullPointerException in HiveAggregate.isBucketedInput()
zhuwei created HIVE-19202: - Summary: CBO failed due to NullPointerException in HiveAggregate.isBucketedInput() Key: HIVE-19202 URL: https://issues.apache.org/jira/browse/HIVE-19202 Project: Hive Issue Type: Bug Reporter: zhuwei Assignee: zhuwei I ran a query with join and group by with below settings, COB failed due to NullPointerException in HiveAggregate.isBucketedInput() set hive.execution.engine=tez; set hive.cbo.costmodel.extended=true; In class HiveRelMdDistribution, we implemented below functions: public RelDistribution distribution(HiveAggregate aggregate, RelMetadataQuery mq) public RelDistribution distribution(HiveJoin join, RelMetadataQuery mq) But in HiveAggregate.isBucketedInput, the argument passed to distribution is "this.getInput()" , obviously it's not right here. The right argument needed is "this" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
zhuwei created HIVE-18871: - Summary: hive on tez execution error due to set hive.aux.jars.path to hdfs:// Key: HIVE-18871 URL: https://issues.apache.org/jira/browse/HIVE-18871 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 2.2.1 Reporter: zhuwei Assignee: zhuwei When set the properties hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar and hive.execution.engine=tez; execute any query will fail with below error log: exec.Task: Failed to execute tez graph. java.lang.IllegalArgumentException: Wrong FS: hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) ~[hadoop-common-2.6.0.jar:?] at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) ~[hadoop-common-2.6.0.jar:?] at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) ~[hadoop-common-2.6.0.jar:?] at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) ~[hadoop-common-2.6.0.jar:?] at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) ~[hadoop-common-2.6.0.jar:?] at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) ~[hadoop-common-2.6.0.jar:?] at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) ~[hadoop-common-2.6.0.jar:?] at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) ~[hadoop-common-2.6.0.jar:?] at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) ~[hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) ~[hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) ~[hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) ~[hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) ~[hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) ~[hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) ~[hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) ~[hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) [hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) [hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) [hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) [hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) [hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) [hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) [hive-exec-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) [hive-cli-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) [hive-cli-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) [hive-cli-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) [hive-cli-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) [hive-cli-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) [hive-cli-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) [hive-cli-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) [hive-cli-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) [hive-cli-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) [hive-cli-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) [hive-cli-2.1.1.jar:2.1.1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111] at org.apache.hadoop.util.RunJar.run(RunJar.java:221) [hadoop-common-2.6.0.jar:?] at
