[jira] [Updated] (HIVE-22082) SQL 错误 [2] [08S01]: Error while processing statement: FAILED: Execution Error
[
https://issues.apache.org/jira/browse/HIVE-22082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
冯伟 updated HIVE-22082:
--
Fix Version/s: 1.2.1
> SQL 错误 [2] [08S01]: Error while processing statement: FAILED: Execution Error
> -
>
> Key: HIVE-22082
> URL: https://issues.apache.org/jira/browse/HIVE-22082
> Project: Hive
> Issue Type: Bug
> Components: Hive, hpl/sql, SQL
>Affects Versions: 3.1.0
> Environment: 报错环境为hive的3.1.0.3.1.0.0-78，可以运行的环境是 Hive 1.2.1000
>Reporter: 冯伟
>Priority: Blocker
> Fix For: 1.2.1
>
>
> hive 1.2.1旧版本SQL语句运行没问题，3.1版本运行报错：
> SQL语句：
> {code:java}
> //代码占位符
> 选择id，从ti_ins.instinct_result_info侧面视图爆炸触发（split（concat_ws（'，'，rule_triggered_1，rule_triggered_2），'，'））num作为触发LIMIT
> 1{code}
> {code:java}
> //代码占位符SQL 错误 [2] [08S01]: Error while processing statement: FAILED:
> Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
> vertexId=vertex_1563265369069_0167_15_00, diagnostics=[Task failed,
> taskId=task_1563265369069_0167_15_00_03, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1563265369069_0167_15_00_03_0:java.lang.RuntimeException:
> java.lang.RuntimeException: java.io.IOException:
> java.lang.IllegalArgumentException: bucketId out of range: -1
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: java.io.IOException:
> java.lang.IllegalArgumentException: bucketId out of range: -1
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:157)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:83)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 16 more
> Caused by: java.io.IOException: java.lang.IllegalArgumentException: bucketId
> out of range: -1
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:421)
> at org.apache.hadoop.mapred.split.TezGroupedSpl
[jira] [Updated] (HIVE-22082) SQL 错误 [2] [08S01]: Error while processing statement: FAILED: Execution Error
[
https://issues.apache.org/jira/browse/HIVE-22082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
冯伟 updated HIVE-22082:
--
Description:
hive 1.2.1旧版本SQL语句运行没问题，3.1版本运行报错：
SQL语句：
{code:java}
// SQL语句：
select id ,triggered from ti_ins.instinct_result_info lateral view
explode(split( concat_ws(',',rule_triggered_1,rule_triggered_2),','))num as
triggered LIMIT 1{code}
{code:java}
//代码占位符SQL 错误 [2] [08S01]: Error while processing statement: FAILED: Execution
Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex
failed, vertexName=Map 1, vertexId=vertex_1563265369069_0167_15_00,
diagnostics=[Task failed, taskId=task_1563265369069_0167_15_00_03,
diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
failure ) :
attempt_1563265369069_0167_15_00_03_0:java.lang.RuntimeException:
java.lang.RuntimeException: java.io.IOException:
java.lang.IllegalArgumentException: bucketId out of range: -1
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.io.IOException:
java.lang.IllegalArgumentException: bucketId out of range: -1
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
at
org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:157)
at org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:83)
at
org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
at org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
at
org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
... 16 more
Caused by: java.io.IOException: java.lang.IllegalArgumentException: bucketId
out of range: -1
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:421)
at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat $
TezGroupedSplitsRecordReader.initNextRecordReader（TezGroupedSplitsInputFormat.java:203）
..还有27个{code}
was:
hive 1.2.1旧版本SQL语句运行没问题，3.1版本运行报错：
SQL语句：
{code:java}
//代码占位符
选择id，从ti_ins.instinct_result_info侧面视图爆炸触发（split（concat_ws（'，'，rule_triggered_1，rule_triggered_2），'，'））num作为触发LIMIT
1{code}
{code:java}
//代码占位符SQL 错误 [2] [08S01]: Error while processing statement: FAILED: Execution
Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex
failed, vertexName=Map 1, vertexId=vertex_1563265369069_0167_15_00,
diagnostics=[Task failed, taskId=task_1563265369069_0167_15_00_03,
diagnostics=[TaskAttempt 0 failed, info=[Err
[jira] [Updated] (HIVE-22082) SQL 错误 [2] [08S01]: Error while processing statement: FAILED: Execution Error
[
https://issues.apache.org/jira/browse/HIVE-22082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
冯伟 updated HIVE-22082:
--
Component/s: Tez
> SQL 错误 [2] [08S01]: Error while processing statement: FAILED: Execution Error
> -
>
> Key: HIVE-22082
> URL: https://issues.apache.org/jira/browse/HIVE-22082
> Project: Hive
> Issue Type: Bug
> Components: Hive, hpl/sql, SQL, Tez
>Affects Versions: 3.1.0
> Environment: 报错环境为hive的3.1.0.3.1.0.0-78，可以运行的环境是 Hive 1.2.1000
>Reporter: 冯伟
>Priority: Blocker
> Fix For: 1.2.1
>
>
> hive 1.2.1旧版本SQL语句运行没问题，3.1版本运行报错：
> SQL语句：
> {code:java}
> // SQL语句：
> select id ,triggered from ti_ins.instinct_result_info lateral view
> explode(split( concat_ws(',',rule_triggered_1,rule_triggered_2),','))num as
> triggered LIMIT 1{code}
> {code:java}
> //代码占位符SQL 错误 [2] [08S01]: Error while processing statement: FAILED:
> Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
> vertexId=vertex_1563265369069_0167_15_00, diagnostics=[Task failed,
> taskId=task_1563265369069_0167_15_00_03, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1563265369069_0167_15_00_03_0:java.lang.RuntimeException:
> java.lang.RuntimeException: java.io.IOException:
> java.lang.IllegalArgumentException: bucketId out of range: -1
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: java.io.IOException:
> java.lang.IllegalArgumentException: bucketId out of range: -1
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:157)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:83)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 16 more
> Caused by: java.io.IOException: java.lang.IllegalArgumentException: bucketId
> out of range: -1
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:42
[jira] [Updated] (HIVE-22082) SQL 错误 [2] [08S01]: Error while processing statement: FAILED: Execution Error
[
https://issues.apache.org/jira/browse/HIVE-22082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
冯伟 updated HIVE-22082:
--
Description:
hive 1.2.1旧版本SQL语句运行没问题，3.1版本运行报错：
查询使用的是TEZ。
SQL语句：
{code:java}
// SQL语句：
select id ,triggered from ti_ins.instinct_result_info lateral view
explode(split( concat_ws(',',rule_triggered_1,rule_triggered_2),','))num as
triggered LIMIT 1{code}
{code:java}
//代码占位符SQL 错误 [2] [08S01]: Error while processing statement: FAILED: Execution
Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex
failed, vertexName=Map 1, vertexId=vertex_1563265369069_0167_15_00,
diagnostics=[Task failed, taskId=task_1563265369069_0167_15_00_03,
diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
failure ) :
attempt_1563265369069_0167_15_00_03_0:java.lang.RuntimeException:
java.lang.RuntimeException: java.io.IOException:
java.lang.IllegalArgumentException: bucketId out of range: -1
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.io.IOException:
java.lang.IllegalArgumentException: bucketId out of range: -1
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
at
org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:157)
at org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:83)
at
org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
at org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
at
org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
... 16 more
Caused by: java.io.IOException: java.lang.IllegalArgumentException: bucketId
out of range: -1
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:421)
at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat $
TezGroupedSplitsRecordReader.initNextRecordReader（TezGroupedSplitsInputFormat.java:203）
..还有27个{code}
was:
hive 1.2.1旧版本SQL语句运行没问题，3.1版本运行报错：
SQL语句：
{code:java}
// SQL语句：
select id ,triggered from ti_ins.instinct_result_info lateral view
explode(split( concat_ws(',',rule_triggered_1,rule_triggered_2),','))num as
triggered LIMIT 1{code}
{code:java}
//代码占位符SQL 错误 [2] [08S01]: Error while processing statement: FAILED: Execution
Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex
failed, vertexName=Map 1, vertexId=vertex_1563265369069_0167_15_00,
diagnostics=[Task failed, taskId=task_1563265369069_016
[jira] [Updated] (HIVE-21828) Tez: Use a pre-parsed TezConfiguration from DagUtils
[ https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-21828: - Status: Open (was: Patch Available) > Tez: Use a pre-parsed TezConfiguration from DagUtils > > > Key: HIVE-21828 > URL: https://issues.apache.org/jira/browse/HIVE-21828 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21828.1.patch, HIVE-21828.2.patch, > HIVE-21828.5.patch, HIVE-21828.6.patch, HIVE-21828.7.patch > > > The HS2 tez-site.xml does not change dynamically - the XML parsed components > of the config can be obtained statically and kept across sessions. > This allows for the replacing of "new TezConfiguration()" with a HS2 local > version instead. > The configuration object however has to reference the right resource file > (i.e location of tez-site.xml) without reparsing it for each query. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-21828) Tez: Use a pre-parsed TezConfiguration from DagUtils
[ https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-21828: - Attachment: HIVE-21828.7.patch > Tez: Use a pre-parsed TezConfiguration from DagUtils > > > Key: HIVE-21828 > URL: https://issues.apache.org/jira/browse/HIVE-21828 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21828.1.patch, HIVE-21828.2.patch, > HIVE-21828.5.patch, HIVE-21828.6.patch, HIVE-21828.7.patch > > > The HS2 tez-site.xml does not change dynamically - the XML parsed components > of the config can be obtained statically and kept across sessions. > This allows for the replacing of "new TezConfiguration()" with a HS2 local > version instead. > The configuration object however has to reference the right resource file > (i.e location of tez-site.xml) without reparsing it for each query. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-21828) Tez: Use a pre-parsed TezConfiguration from DagUtils
[ https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-21828: - Status: Patch Available (was: Open) > Tez: Use a pre-parsed TezConfiguration from DagUtils > > > Key: HIVE-21828 > URL: https://issues.apache.org/jira/browse/HIVE-21828 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21828.1.patch, HIVE-21828.2.patch, > HIVE-21828.5.patch, HIVE-21828.6.patch, HIVE-21828.7.patch > > > The HS2 tez-site.xml does not change dynamically - the XML parsed components > of the config can be obtained statically and kept across sessions. > This allows for the replacing of "new TezConfiguration()" with a HS2 local > version instead. > The configuration object however has to reference the right resource file > (i.e location of tez-site.xml) without reparsing it for each query. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-21828) Tez: Use a pre-parsed TezConfiguration from DagUtils
[ https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-21828: - Attachment: (was: HIVE-21828.7.patch) > Tez: Use a pre-parsed TezConfiguration from DagUtils > > > Key: HIVE-21828 > URL: https://issues.apache.org/jira/browse/HIVE-21828 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21828.1.patch, HIVE-21828.2.patch, > HIVE-21828.5.patch, HIVE-21828.6.patch, HIVE-21828.7.patch > > > The HS2 tez-site.xml does not change dynamically - the XML parsed components > of the config can be obtained statically and kept across sessions. > This allows for the replacing of "new TezConfiguration()" with a HS2 local > version instead. > The configuration object however has to reference the right resource file > (i.e location of tez-site.xml) without reparsing it for each query. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21828) Tez: Use a pre-parsed TezConfiguration from DagUtils
[
https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899905#comment-16899905
]
Hive QA commented on HIVE-21828:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18255/dev-support/hive-personality.sh
|
| git revision | master / 212b428 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18255/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Tez: Use a pre-parsed TezConfiguration from DagUtils
>
>
> Key: HIVE-21828
> URL: https://issues.apache.org/jira/browse/HIVE-21828
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21828.1.patch, HIVE-21828.2.patch,
> HIVE-21828.5.patch, HIVE-21828.6.patch, HIVE-21828.7.patch
>
>
> The HS2 tez-site.xml does not change dynamically - the XML parsed components
> of the config can be obtained statically and kept across sessions.
> This allows for the replacing of "new TezConfiguration()" with a HS2 local
> version instead.
> The configuration object however has to reference the right resource file
> (i.e location of tez-site.xml) without reparsing it for each query.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21828) Tez: Use a pre-parsed TezConfiguration from DagUtils
[
https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899947#comment-16899947
]
Hive QA commented on HIVE-21828:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976669/HIVE-21828.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16723 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_fixed_bucket_pruning]
(batchId=182)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18255/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18255/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18255/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976669 - PreCommit-HIVE-Build
> Tez: Use a pre-parsed TezConfiguration from DagUtils
>
>
> Key: HIVE-21828
> URL: https://issues.apache.org/jira/browse/HIVE-21828
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21828.1.patch, HIVE-21828.2.patch,
> HIVE-21828.5.patch, HIVE-21828.6.patch, HIVE-21828.7.patch
>
>
> The HS2 tez-site.xml does not change dynamically - the XML parsed components
> of the config can be obtained statically and kept across sessions.
> This allows for the replacing of "new TezConfiguration()" with a HS2 local
> version instead.
> The configuration object however has to reference the right resource file
> (i.e location of tez-site.xml) without reparsing it for each query.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work started] (HIVE-22083) Values of tag order cannot be null, so it can be "byte" instead of "Byte"
[ https://issues.apache.org/jira/browse/HIVE-22083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22083 started by Ivan Suller. -- > Values of tag order cannot be null, so it can be "byte" instead of "Byte" > - > > Key: HIVE-22083 > URL: https://issues.apache.org/jira/browse/HIVE-22083 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > > Values of tag order cannot be null, so it can be "byte" instead of "Byte". > Switching between Byte and byte is "cheap" - the Byte objects are cached by > the JVM - but it still costs a bit more memory and CPU usage. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-22083) Values of tag order cannot be null, so it can be "byte" instead of "Byte"
[ https://issues.apache.org/jira/browse/HIVE-22083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller reassigned HIVE-22083: -- Assignee: Ivan Suller > Values of tag order cannot be null, so it can be "byte" instead of "Byte" > - > > Key: HIVE-22083 > URL: https://issues.apache.org/jira/browse/HIVE-22083 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > > Values of tag order cannot be null, so it can be "byte" instead of "Byte". > Switching between Byte and byte is "cheap" - the Byte objects are cached by > the JVM - but it still costs a bit more memory and CPU usage. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22040) Drop partition throws exception with 'Failed to delete parent: File does not exist' when the partition's parent path does not exists
[
https://issues.apache.org/jira/browse/HIVE-22040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiepengjie updated HIVE-22040:
--
Attachment: HIVE-22040.03.patch
> Drop partition throws exception with 'Failed to delete parent: File does not
> exist' when the partition's parent path does not exists
>
>
> Key: HIVE-22040
> URL: https://issues.apache.org/jira/browse/HIVE-22040
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: xiepengjie
>Assignee: xiepengjie
>Priority: Major
> Attachments: HIVE-22040.01.patch, HIVE-22040.02.patch,
> HIVE-22040.03.patch, HIVE-22040.patch
>
>
> I create a manage table with multi partition columns, when i try to drop
> partition throws exception with 'Failed to delete parent: File does not
> exist' when the partition's parent path does not exist. The partition's
> metadata in mysql has been deleted, but the exception is still thrown. it
> will fail if connecting hiveserver2 with jdbc by java, this problem also
> exists in master branch, I think it is very unfriendly and we should fix it.
> Example:
> – First, create manage table with nulti partition columns, and add partitions:
> {code:java}
> drop table if exists t1;
> create table t1 (c1 int) partitioned by (year string, month string, day
> string);
> alter table t1 add partition(year='2019', month='07', day='01');{code}
> – Second, delete the path of partition 'month=07':
> {code:java}
> hadoop fs -rm -r
> /user/hadoop/xiepengjietest.db/drop_partition/year=2019/month=07{code}
> -- Third, when i try to drop partition, the metastore throws exception with
> 'Failed to delete parent: File does not exist' .
> {code:java}
> alter table t1 drop partition(year='2019', month='07', day='01');
> {code}
> exception like this:
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Failed to delete parent: File
> does not exist:
> /user/hadoop/xiepengjietest.db/drop_partition/year=2019/month=07
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getContentSummaryInt(FSDirStatAndListingOp.java:493)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getContentSummary(FSDirStatAndListingOp.java:140)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getContentSummary(FSNamesystem.java:3995)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getContentSummary(NameNodeRpcServer.java:1202)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getContentSummary(ClientNamenodeProtocolServerSideTranslatorPB.java:883)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22077) Inserting overwrite partitions clause does not clean directories while partitions' info is not stored in metadata
[
https://issues.apache.org/jira/browse/HIVE-22077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui An updated HIVE-22077:
--
Description:
Inserting overwrite static partitions may not clean related HDFS location if
partitions' info is not stored in metadata.
Steps to reproduce this issue :
1. Create a managed table :
{code:sql}
CREATE TABLE `test`(
`id` string)
PARTITIONED BY (
`dayno` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION |
'hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test'
TBLPROPERTIES (
'transient_lastDdlTime'='1564731656')
{code}
2. Create partition's directory and put some data in it
{code:java}
hdfs dfs -mkdir
hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
hdfs dfs -put test.data
hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
{code}
3. Insert overwrite partition dayno=20190802
{code:sql}
INSERT OVERWRITE TABLE test PARTITION(dayno='20190802')
SELECT "some value";
{code}
4. We could see the test.data under partition directory is not deleted.
was:
Inserting overwrite static partitions may not clean related HDFS location if
partitions' info is not stored in metadata.
Steps to Reproduce this issue :
1. Create a managed table :
{code:sql}
CREATE TABLE `test`(
`id` string)
PARTITIONED BY (
`dayno` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION |
'hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test'
TBLPROPERTIES (
'transient_lastDdlTime'='1564731656')
{code}
2. Create partition's directory and put some data under it
{code:java}
hdfs dfs -mkdir
hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
hdfs dfs -put test.data
hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
{code}
3. Insert overwrite partition dayno=20190802
{code:sql}
INSERT OVERWRITE TABLE test PARTITION(dayno='20190802')
SELECT "some value";
{code}
4. We could see the test.data under partition directory is not deleted.
> Inserting overwrite partitions clause does not clean directories while
> partitions' info is not stored in metadata
> -
>
> Key: HIVE-22077
> URL: https://issues.apache.org/jira/browse/HIVE-22077
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.1, 4.0.0, 2.3.4
>Reporter: Hui An
>Assignee: Hui An
>Priority: Major
>
> Inserting overwrite static partitions may not clean related HDFS location if
> partitions' info is not stored in metadata.
> Steps to reproduce this issue :
>
> 1. Create a managed table :
>
> {code:sql}
> CREATE TABLE `test`(
>`id` string)
> PARTITIONED BY (
>`dayno` string)
> ROW FORMAT SERDE
>'org.apache.hadoop.hive
[jira] [Updated] (HIVE-21875) Implement drop partition related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-21875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-21875:
-
Attachment: HIVE-21875.01.patch
> Implement drop partition related methods on temporary tables
>
>
> Key: HIVE-21875
> URL: https://issues.apache.org/jira/browse/HIVE-21875
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21875.01.patch
>
>
> IMetaStoreClient exposes the following methods related to dropping partitions:
> {code:java}
> boolean dropPartition(String db_name, String tbl_name, List
> part_vals, boolean deleteData);
> boolean dropPartition(String catName, String db_name, String tbl_name,
> List part_vals, boolean deleteData);
> boolean dropPartition(String db_name, String tbl_name, List
> part_vals, PartitionDropOptions options);
> boolean dropPartition(String catName, String db_name, String tbl_name,
> List part_vals, PartitionDropOptions options);
> List dropPartitions(String dbName, String tblName,
> List> partExprs, boolean deleteData, boolean
> ifExists);
> List dropPartitions(String catName, String dbName, String tblName,
> List> partExprs, boolean deleteData, boolean
> ifExists);
> List dropPartitions(String dbName, String tblName,
> List> partExprs, boolean deleteData, boolean
> ifExists, boolean needResults);
> List dropPartitions(String catName, String dbName, String tblName,
> List> partExprs, boolean deleteData, boolean
> ifExists, boolean needResults);
> List dropPartitions(String dbName, String tblName,
> List> partExprs, PartitionDropOptions options);
> List dropPartitions(String catName, String dbName, String tblName,
> List> partExprs, PartitionDropOptions options);
> boolean dropPartition(String db_name, String tbl_name, String name, boolean
> deleteData);
> boolean dropPartition(String catName, String db_name, String tbl_name, String
> name, boolean deleteData){code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-21875) Implement drop partition related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-21875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-21875:
-
Status: Patch Available (was: Open)
> Implement drop partition related methods on temporary tables
>
>
> Key: HIVE-21875
> URL: https://issues.apache.org/jira/browse/HIVE-21875
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21875.01.patch
>
>
> IMetaStoreClient exposes the following methods related to dropping partitions:
> {code:java}
> boolean dropPartition(String db_name, String tbl_name, List
> part_vals, boolean deleteData);
> boolean dropPartition(String catName, String db_name, String tbl_name,
> List part_vals, boolean deleteData);
> boolean dropPartition(String db_name, String tbl_name, List
> part_vals, PartitionDropOptions options);
> boolean dropPartition(String catName, String db_name, String tbl_name,
> List part_vals, PartitionDropOptions options);
> List dropPartitions(String dbName, String tblName,
> List> partExprs, boolean deleteData, boolean
> ifExists);
> List dropPartitions(String catName, String dbName, String tblName,
> List> partExprs, boolean deleteData, boolean
> ifExists);
> List dropPartitions(String dbName, String tblName,
> List> partExprs, boolean deleteData, boolean
> ifExists, boolean needResults);
> List dropPartitions(String catName, String dbName, String tblName,
> List> partExprs, boolean deleteData, boolean
> ifExists, boolean needResults);
> List dropPartitions(String dbName, String tblName,
> List> partExprs, PartitionDropOptions options);
> List dropPartitions(String catName, String dbName, String tblName,
> List> partExprs, PartitionDropOptions options);
> boolean dropPartition(String db_name, String tbl_name, String name, boolean
> deleteData);
> boolean dropPartition(String catName, String db_name, String tbl_name, String
> name, boolean deleteData){code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (HIVE-22084) Implement exchange partitions related methods on SessionHiveMetastoreClient
[
https://issues.apache.org/jira/browse/HIVE-22084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter reassigned HIVE-22084:
> Implement exchange partitions related methods on SessionHiveMetastoreClient
> ---
>
> Key: HIVE-22084
> URL: https://issues.apache.org/jira/browse/HIVE-22084
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
>
> IMetaStoreClient exposes the following methods related to exchanging
> partitions:
> {code:java}
> Partition exchange_partition(Map partitionSpecs, String
> sourceDb, String sourceTable, String destdb, String destTableName);
> Partition exchange_partition(Map partitionSpecs, String
> sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceDb, String sourceTable, String destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);{code}
> In order to support partitions on temporary tables, these methods must be
> implemented.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22084) Implement exchange partitions related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22084:
-
Summary: Implement exchange partitions related methods on temporary tables
(was: Implement exchange partitions related methods on
SessionHiveMetastoreClient)
> Implement exchange partitions related methods on temporary tables
> -
>
> Key: HIVE-22084
> URL: https://issues.apache.org/jira/browse/HIVE-22084
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
>
> IMetaStoreClient exposes the following methods related to exchanging
> partitions:
> {code:java}
> Partition exchange_partition(Map partitionSpecs, String
> sourceDb, String sourceTable, String destdb, String destTableName);
> Partition exchange_partition(Map partitionSpecs, String
> sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceDb, String sourceTable, String destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);{code}
> In order to support partitions on temporary tables, these methods must be
> implemented.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22083) Values of tag order cannot be null, so it can be "byte" instead of "Byte"
[ https://issues.apache.org/jira/browse/HIVE-22083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-22083: --- Attachment: HIVE-22083.1.patch > Values of tag order cannot be null, so it can be "byte" instead of "Byte" > - > > Key: HIVE-22083 > URL: https://issues.apache.org/jira/browse/HIVE-22083 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22083.1.patch > > > Values of tag order cannot be null, so it can be "byte" instead of "Byte". > Switching between Byte and byte is "cheap" - the Byte objects are cached by > the JVM - but it still costs a bit more memory and CPU usage. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-19535) Hive error reporting for not setting “Utilities.INPUT_NAME”
[ https://issues.apache.org/jira/browse/HIVE-19535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900076#comment-16900076 ] Hui An commented on HIVE-19535: --- Could you please give related cases and detailed error logs? > Hive error reporting for not setting “Utilities.INPUT_NAME” > --- > > Key: HIVE-19535 > URL: https://issues.apache.org/jira/browse/HIVE-19535 > Project: Hive > Issue Type: Bug >Reporter: zhaoyang >Priority: Blocker > > I set the parameter "hive.exec.mode.local.auto" to true. Then I used JDBC > mode to submit sql to hiveserver2 in a multithreaded manner. Sometimes > exceptions are thrown when several tasks run in parallel in the local mode. > "Invalid input path xxx" > But this path is not the input path of the current SQL, it is another SQL > input path. > So I suspect that one of the SQL covers some information about another SQL. > So I looked up the source code and found a parameter, and when I set the > "iocontext.input.name" to the unique value, I didn't get it wrong. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22040) Drop partition throws exception with 'Failed to delete parent: File does not exist' when the partition's parent path does not exists
[
https://issues.apache.org/jira/browse/HIVE-22040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900081#comment-16900081
]
Hive QA commented on HIVE-22040:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976692/HIVE-22040.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18256/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18256/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18256/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-08-05 13:26:12.682
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-18256/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-08-05 13:26:12.685
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 212b428 HIVE-22054: Avoid recursive listing to check if a
directory is empty (Prabhas Kumar Samanta, reviewed by Steve Loughran and Jason
Dere)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 212b428 HIVE-22054: Avoid recursive listing to check if a
directory is empty (Prabhas Kumar Samanta, reviewed by Steve Loughran and Jason
Dere)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-08-05 13:26:13.999
+ rm -rf ../yetus_PreCommit-HIVE-Build-18256
+ mkdir ../yetus_PreCommit-HIVE-Build-18256
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-18256
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-18256/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:374
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java'
with conflicts.
Going to apply patch with: git apply -p0
error: patch failed:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:374
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java'
with conflicts.
U
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-18256
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976692 - PreCommit-HIVE-Build
> Drop partition throws exception with 'Failed to delete parent: File does not
> exist' when the partition's parent path does not exists
>
>
> Key: HIVE-22040
> URL: https://issues.apache.org/jira/browse/HIVE-22040
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: xiepengjie
>Assignee: xiepengjie
>Priority: Major
> Attachments: HIVE-22040.01.patch, HIVE-22040.02.patch,
> HIVE-22040.03.patch, HIVE-22040.patch
>
>
> I create a manage table with multi partition columns, when i try to drop
> partition throws exception with 'Failed to delete parent: File d
[jira] [Assigned] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[ https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-16690: --- Assignee: Laszlo Bodor (was: Zhiyuan Yang) > Configure Tez cartesian product edge based on LLAP cluster size > --- > > Key: HIVE-16690 > URL: https://issues.apache.org/jira/browse/HIVE-16690 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-16690.1.patch, HIVE-16690.2.patch, > HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.2.patch, > HIVE-16690.2.patch, HIVE-16690.addendum.patch > > > In HIVE-14731 we are using default value for target parallelism of fair > cartesian product edge. Ideally this should be set according to cluster size. > In case of LLAP it's pretty easy to get cluster size, i.e., number of > executors. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[ https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-16690: Attachment: HIVE-16690.2.patch > Configure Tez cartesian product edge based on LLAP cluster size > --- > > Key: HIVE-16690 > URL: https://issues.apache.org/jira/browse/HIVE-16690 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-16690.1.patch, HIVE-16690.2.patch, > HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.2.patch, > HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.addendum.patch > > > In HIVE-14731 we are using default value for target parallelism of fair > cartesian product edge. Ideally this should be set according to cluster size. > In case of LLAP it's pretty easy to get cluster size, i.e., number of > executors. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-20801: --- Assignee: Gopal V (was: Laszlo Bodor) > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-20801: --- Assignee: Laszlo Bodor (was: Gopal V) > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Laszlo Bodor >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-20801: Attachment: HIVE-20801.3.patch > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Laszlo Bodor >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-21828) Tez: Use a pre-parsed TezConfiguration from DagUtils
[ https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-21828: - Attachment: HIVE-21828.8.patch > Tez: Use a pre-parsed TezConfiguration from DagUtils > > > Key: HIVE-21828 > URL: https://issues.apache.org/jira/browse/HIVE-21828 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21828.1.patch, HIVE-21828.2.patch, > HIVE-21828.5.patch, HIVE-21828.6.patch, HIVE-21828.7.patch, HIVE-21828.8.patch > > > The HS2 tez-site.xml does not change dynamically - the XML parsed components > of the config can be obtained statically and kept across sessions. > This allows for the replacing of "new TezConfiguration()" with a HS2 local > version instead. > The configuration object however has to reference the right resource file > (i.e location of tez-site.xml) without reparsing it for each query. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-4605:
---
Status: In Progress (was: Patch Available)
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.3.0, 0.13.1, 0.13.0, 0.12.0, 0.11.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21875) Implement drop partition related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-21875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900157#comment-16900157
]
Hive QA commented on HIVE-21875:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
11s{color} | {color:blue} standalone-metastore/metastore-server in master has
180 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 1 unchanged - 1 fixed = 1 total (was 2) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 2 new + 15 unchanged - 0 fixed
= 17 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
18s{color} | {color:red} ql generated 1 new + 2250 unchanged - 0 fixed = 2251
total (was 2250) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Null passed for non-null parameter of getPartitionedTempTable(Table) in
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.dropPartition(String,
String, String, String, boolean) Method invoked at
SessionHiveMetaStoreClient.java:of getPartitionedTempTable(Table) in
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.dropPartition(String,
String, String, String, boolean) Method invoked at
SessionHiveMetaStoreClient.java:[line 1553] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18257/dev-support/hive-personality.sh
|
| git revision | master / 212b428 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18257/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18257/yetus/new-findbugs-ql.html
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18257/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement drop partition related methods on temporary tables
>
>
>
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-4605:
---
Attachment: HIVE-4605.3.patch
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.3.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-4605:
---
Status: Patch Available (was: In Progress)
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.3.0, 0.13.1, 0.13.0, 0.12.0, 0.11.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.3.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900162#comment-16900162
]
Oleksiy Sayankin commented on HIVE-4605:
Added not null verification for {{finalPaths[idx]}}.
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.3.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21875) Implement drop partition related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-21875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900215#comment-16900215
]
Hive QA commented on HIVE-21875:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976695/HIVE-21875.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 16795 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_1_drop] (batchId=13)
org.apache.hadoop.hive.ql.metadata.TestHive.testAutoPurgeTablesAndPartitions
(batchId=329)
org.apache.hadoop.hive.ql.metadata.TestHive.testDropPartitionsWithPurge
(batchId=329)
org.apache.hadoop.hive.ql.metadata.TestHiveRemote.testAutoPurgeTablesAndPartitions
(batchId=330)
org.apache.hadoop.hive.ql.metadata.TestHiveRemote.testDropPartitionsWithPurge
(batchId=330)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testIncrementalReplWithEventsBatchHavingDropCreateTable
(batchId=259)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testReplLoadFromSourceUsingWithClause
(batchId=259)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testDropPartitionEventWithPartitionOnTimestampColumn
(batchId=251)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testDrops
(batchId=251)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testDropsWithCM
(batchId=251)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testExchangePartition
(batchId=251)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testIncrementalRepeatEventOnMissingObject
(batchId=251)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testStatus
(batchId=251)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testWithStringPartitionSpecialChars
(batchId=251)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testDropPartitionEventWithPartitionOnTimestampColumn
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testDrops (batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testDropsWithCM
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testExchangePartition
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalRepeatEventOnMissingObject
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testWithStringPartitionSpecialChars
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testIncrementalReplWithEventsBatchHavingDropCreateTable
(batchId=263)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testReplLoadFromSourceUsingWithClause
(batchId=263)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18257/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18257/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18257/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976695 - PreCommit-HIVE-Build
> Implement drop partition related methods on temporary tables
>
>
> Key: HIVE-21875
> URL: https://issues.apache.org/jira/browse/HIVE-21875
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21875.01.patch
>
>
> IMetaStoreClient exposes the following methods related to dropping partitions:
> {code:java}
> boolean dropPartition(String db_name, String tbl_name, List
> part_vals, boolean deleteData);
> boolean dropPartition(String catName, String db_name, String tbl_name,
> List part_vals, boolean deleteData);
> boolean dropPartition(String db_name, String tbl_name, List
> part_vals, PartitionDropOptions options);
> boolean dropPartition(String catName, String db_name, String tbl_name,
> List part_vals, PartitionDropOptions options);
> List dropPartitions(String dbName, String tblName,
> List> partExprs, boolean deleteData, boolean
> ifExists);
> List dropPartitions(String catName, String dbName, String tblName,
> List> partExprs, boolean deleteData, boolean
> ifExists);
> List dropPartitions(String dbName, String tblName,
> List>
[jira] [Commented] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[
https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900247#comment-16900247
]
Hive QA commented on HIVE-16690:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 3 new + 36 unchanged - 0 fixed
= 39 total (was 36) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18258/dev-support/hive-personality.sh
|
| git revision | master / 212b428 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18258/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18258/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Configure Tez cartesian product edge based on LLAP cluster size
> ---
>
> Key: HIVE-16690
> URL: https://issues.apache.org/jira/browse/HIVE-16690
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-16690.1.patch, HIVE-16690.2.patch,
> HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.2.patch,
> HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.addendum.patch
>
>
> In HIVE-14731 we are using default value for target parallelism of fair
> cartesian product edge. Ideally this should be set according to cluster size.
> In case of LLAP it's pretty easy to get cluster size, i.e., number of
> executors.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22079:
---
Status: Open (was: Patch Available)
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22079:
---
Attachment: HIVE-22079.2.patch
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22079:
---
Status: Patch Available (was: Open)
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[
https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900282#comment-16900282
]
Hive QA commented on HIVE-16690:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976704/HIVE-16690.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 16691 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=232)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=232)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_mapjoin3]
(batchId=163)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18258/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18258/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18258/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976704 - PreCommit-HIVE-Build
> Configure Tez cartesian product edge based on LLAP cluster size
> ---
>
> Key: HIVE-16690
> URL: https://issues.apache.org/jira/browse/HIVE-16690
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-16690.1.patch, HIVE-16690.2.patch,
> HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.2.patch,
> HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.addendum.patch
>
>
> In HIVE-14731 we are using default value for target parallelism of fair
> cartesian product edge. Ideally this should be set according to cluster size.
> In case of LLAP it's pretty easy to get cluster size, i.e., number of
> executors.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (HIVE-22046) Differentiate among column stats computed by different engines
[
https://issues.apache.org/jira/browse/HIVE-22046?focusedWorklogId=289091&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289091
]
ASF GitHub Bot logged work on HIVE-22046:
-
Author: ASF GitHub Bot
Created on: 05/Aug/19 18:06
Start Date: 05/Aug/19 18:06
Worklog Time Spent: 10m
Work Description: jdere commented on pull request #741: HIVE-22046
URL: https://github.com/apache/hive/pull/741#discussion_r310720963
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -3255,14 +3255,14 @@ public Table get_table_core(
final String name,
final String writeIdList)
throws MetaException, NoSuchObjectException {
- return get_table_core(catName, dbname, name, writeIdList, false);
+ return get_table_core(catName, dbname, name, writeIdList, false, null);
Review comment:
same question as above
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 289091)
Time Spent: 0.5h (was: 20m)
> Differentiate among column stats computed by different engines
> --
>
> Key: HIVE-22046
> URL: https://issues.apache.org/jira/browse/HIVE-22046
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22046.01.patch, HIVE-22046.02.patch,
> HIVE-22046.03.patch, HIVE-22046.04.patch, HIVE-22046.05.patch,
> HIVE-22046.06.patch, HIVE-22046.07.patch, HIVE-22046.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The goal is to avoid computation of column stats by engines to step on each
> other, e.g., Hive and Impala. In longer term, we may introduce a common
> representation for the column statistics stored by different engines.
> For this issue, we will add a new column 'engine' to TAB_COL_STATS HMS table
> (unpartitioned tables) and to PART_COL_STATS HMS table (partitioned tables).
> This will prevent conflicts at the column level stats.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (HIVE-22046) Differentiate among column stats computed by different engines
[
https://issues.apache.org/jira/browse/HIVE-22046?focusedWorklogId=289090&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289090
]
ASF GitHub Bot logged work on HIVE-22046:
-
Author: ASF GitHub Bot
Created on: 05/Aug/19 18:06
Start Date: 05/Aug/19 18:06
Worklog Time Spent: 10m
Work Description: jdere commented on pull request #741: HIVE-22046
URL: https://github.com/apache/hive/pull/741#discussion_r310720812
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -3089,7 +3089,7 @@ public Table get_table(final String dbname, final String
name) throws MetaExcept
NoSuchObjectException {
String[] parsedDbName = parseDbName(dbname, conf);
return getTableInternal(
-parsedDbName[CAT_NAME], parsedDbName[DB_NAME], name, null, null,
false);
+parsedDbName[CAT_NAME], parsedDbName[DB_NAME], name, null, null,
false, null);
Review comment:
Should this be null? Does that internally default to "hive" in
getTableColumnStatistics()? Or is this safe because we are not getting column
stats in this case?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 289090)
Time Spent: 20m (was: 10m)
> Differentiate among column stats computed by different engines
> --
>
> Key: HIVE-22046
> URL: https://issues.apache.org/jira/browse/HIVE-22046
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22046.01.patch, HIVE-22046.02.patch,
> HIVE-22046.03.patch, HIVE-22046.04.patch, HIVE-22046.05.patch,
> HIVE-22046.06.patch, HIVE-22046.07.patch, HIVE-22046.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The goal is to avoid computation of column stats by engines to step on each
> other, e.g., Hive and Impala. In longer term, we may introduce a common
> representation for the column statistics stored by different engines.
> For this issue, we will add a new column 'engine' to TAB_COL_STATS HMS table
> (unpartitioned tables) and to PART_COL_STATS HMS table (partitioned tables).
> This will prevent conflicts at the column level stats.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (HIVE-16587) NPE when inserting complex types with nested null values
[
https://issues.apache.org/jira/browse/HIVE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naresh P R reassigned HIVE-16587:
-
Assignee: Naresh P R
> NPE when inserting complex types with nested null values
>
>
> Key: HIVE-16587
> URL: https://issues.apache.org/jira/browse/HIVE-16587
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1
>Reporter: Jason Dere
>Assignee: Naresh P R
>Priority: Major
>
> {noformat}
> CREATE TABLE complex1 (c0 int, c1 array, c2 map, c3
> struct>, c4 array f3:array>>)
> insert into complex1
> select 3, array(1, 2, null), map(1, 'one', 2, null), named_struct('f1',
> cast(null as int), 'f2', cast(null as string), 'f3', array(1,2,null)),
> array(named_struct('f1', 11, 'f2', 'two', 'f3', array(2,3,4)))
> {noformat}
> Gives the following error:
> {noformat}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:207)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:291)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:255)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:531)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
> at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$Proxy126.executeStatementAsync(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:530)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getWritableSize(StatsUtils.java:1144)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getSizeOfMap(StatsUtils.java:1106)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getSizeOfComplexTypes(StatsUtils.java:978)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getAvgColLenOf(StatsUtils.java:916)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExpression(StatsUtils.java:1371)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExprMap(StatsUtils.java:1194)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$SelectStatsRule.process(StatsRulesProcFactory.java:187)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.walk(LevelOrderWalker.java:143)
>
[jira] [Work started] (HIVE-16587) NPE when inserting complex types with nested null values
[
https://issues.apache.org/jira/browse/HIVE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-16587 started by Naresh P R.
-
> NPE when inserting complex types with nested null values
>
>
> Key: HIVE-16587
> URL: https://issues.apache.org/jira/browse/HIVE-16587
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1
>Reporter: Jason Dere
>Assignee: Naresh P R
>Priority: Major
>
> {noformat}
> CREATE TABLE complex1 (c0 int, c1 array, c2 map, c3
> struct>, c4 array f3:array>>)
> insert into complex1
> select 3, array(1, 2, null), map(1, 'one', 2, null), named_struct('f1',
> cast(null as int), 'f2', cast(null as string), 'f3', array(1,2,null)),
> array(named_struct('f1', 11, 'f2', 'two', 'f3', array(2,3,4)))
> {noformat}
> Gives the following error:
> {noformat}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:207)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:291)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:255)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:531)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
> at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$Proxy126.executeStatementAsync(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:530)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getWritableSize(StatsUtils.java:1144)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getSizeOfMap(StatsUtils.java:1106)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getSizeOfComplexTypes(StatsUtils.java:978)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getAvgColLenOf(StatsUtils.java:916)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExpression(StatsUtils.java:1371)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExprMap(StatsUtils.java:1194)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$SelectStatsRule.process(StatsRulesProcFactory.java:187)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.walk(LevelOrderWalker.java:143)
> at
> org.a
[jira] [Updated] (HIVE-16587) NPE when inserting complex types with nested null values
[
https://issues.apache.org/jira/browse/HIVE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naresh P R updated HIVE-16587:
--
Attachment: HIVE-16587.patch
Status: Patch Available (was: In Progress)
> NPE when inserting complex types with nested null values
>
>
> Key: HIVE-16587
> URL: https://issues.apache.org/jira/browse/HIVE-16587
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1
>Reporter: Jason Dere
>Assignee: Naresh P R
>Priority: Major
> Attachments: HIVE-16587.patch
>
>
> {noformat}
> CREATE TABLE complex1 (c0 int, c1 array, c2 map, c3
> struct>, c4 array f3:array>>)
> insert into complex1
> select 3, array(1, 2, null), map(1, 'one', 2, null), named_struct('f1',
> cast(null as int), 'f2', cast(null as string), 'f3', array(1,2,null)),
> array(named_struct('f1', 11, 'f2', 'two', 'f3', array(2,3,4)))
> {noformat}
> Gives the following error:
> {noformat}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:207)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:291)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:255)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:531)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
> at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$Proxy126.executeStatementAsync(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:530)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getWritableSize(StatsUtils.java:1144)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getSizeOfMap(StatsUtils.java:1106)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getSizeOfComplexTypes(StatsUtils.java:978)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getAvgColLenOf(StatsUtils.java:916)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExpression(StatsUtils.java:1371)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExprMap(StatsUtils.java:1194)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$SelectStatsRule.process(StatsRulesProcFactory.java:187)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
>
[jira] [Work logged] (HIVE-22046) Differentiate among column stats computed by different engines
[
https://issues.apache.org/jira/browse/HIVE-22046?focusedWorklogId=289126&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289126
]
ASF GitHub Bot logged work on HIVE-22046:
-
Author: ASF GitHub Bot
Created on: 05/Aug/19 18:37
Start Date: 05/Aug/19 18:37
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #741: HIVE-22046
URL: https://github.com/apache/hive/pull/741#discussion_r310738062
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -3089,7 +3089,7 @@ public Table get_table(final String dbname, final String
name) throws MetaExcept
NoSuchObjectException {
String[] parsedDbName = parseDbName(dbname, conf);
return getTableInternal(
-parsedDbName[CAT_NAME], parsedDbName[DB_NAME], name, null, null,
false);
+parsedDbName[CAT_NAME], parsedDbName[DB_NAME], name, null, null,
false, null);
Review comment:
Observe in this case, retrieve column stats (parameter before null) is
false. Basically, when we retrieve a given table, if we want to retrieve stats,
we will only retrieve stats for a given engine. Since this is not a public API
but rather a private method, I am thinking whether I should eliminate the
boolean and document behavior as 'if engine is specified, retrieve column stats
for that engine; if engine is null, do not retrieve column stats'. What do you
think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 289126)
Time Spent: 40m (was: 0.5h)
> Differentiate among column stats computed by different engines
> --
>
> Key: HIVE-22046
> URL: https://issues.apache.org/jira/browse/HIVE-22046
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22046.01.patch, HIVE-22046.02.patch,
> HIVE-22046.03.patch, HIVE-22046.04.patch, HIVE-22046.05.patch,
> HIVE-22046.06.patch, HIVE-22046.07.patch, HIVE-22046.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The goal is to avoid computation of column stats by engines to step on each
> other, e.g., Hive and Impala. In longer term, we may introduce a common
> representation for the column statistics stored by different engines.
> For this issue, we will add a new column 'engine' to TAB_COL_STATS HMS table
> (unpartitioned tables) and to PART_COL_STATS HMS table (partitioned tables).
> This will prevent conflicts at the column level stats.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (HIVE-22001) AcidUtils.getAcidState() can fail if Cleaner is removing files at the same time
[
https://issues.apache.org/jira/browse/HIVE-22001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere reassigned HIVE-22001:
-
Assignee: Jason Dere
> AcidUtils.getAcidState() can fail if Cleaner is removing files at the same
> time
> ---
>
> Key: HIVE-22001
> URL: https://issues.apache.org/jira/browse/HIVE-22001
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
>
> Had one user hit the following error during getSplits
> {noformat}
> 2019-07-06T14:33:03,067 ERROR [4640181a-3eb7-4b3e-9a40-d7a8de9a570c
> HiveServer2-HttpHandler-Pool: Thread-415519]: SessionState
> (SessionState.java:printError(1247)) - Vertex failed, vertexName=Map 1,
> vertexId=vertex_1560947172646_2452_6199_00, diagnostics=[Vertex
> vertex_1560947172646_2452_6199_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: hive_table initializer failed,
> vertex=vertex_1560947172646_2452_6199_00 [Map 1], java.lang.RuntimeException:
> ORC split generation failed with exception: java.io.FileNotFoundException:
> File hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070
> does not exist.
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1870)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1958)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1809)
> ... 17 more
> Caused by: java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1126)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1953)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$MetaDataFile.chooseFile(AcidUtils.java:1903)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$MetaDataFile.isRawFormat(AcidUtils.java:1913)
> at
> org.apache.hadoop.
[jira] [Work logged] (HIVE-22046) Differentiate among column stats computed by different engines
[
https://issues.apache.org/jira/browse/HIVE-22046?focusedWorklogId=289127&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289127
]
ASF GitHub Bot logged work on HIVE-22046:
-
Author: ASF GitHub Bot
Created on: 05/Aug/19 18:39
Start Date: 05/Aug/19 18:39
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #741: HIVE-22046
URL: https://github.com/apache/hive/pull/741#discussion_r310738647
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -3255,14 +3255,14 @@ public Table get_table_core(
final String name,
final String writeIdList)
throws MetaException, NoSuchObjectException {
- return get_table_core(catName, dbname, name, writeIdList, false);
+ return get_table_core(catName, dbname, name, writeIdList, false, null);
Review comment:
Same as above. `get_table_core` with retrieve column stats is a public
method but not part of the Metastore API, hence I think I can do same as above.
Thoughts?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 289127)
Time Spent: 50m (was: 40m)
> Differentiate among column stats computed by different engines
> --
>
> Key: HIVE-22046
> URL: https://issues.apache.org/jira/browse/HIVE-22046
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22046.01.patch, HIVE-22046.02.patch,
> HIVE-22046.03.patch, HIVE-22046.04.patch, HIVE-22046.05.patch,
> HIVE-22046.06.patch, HIVE-22046.07.patch, HIVE-22046.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The goal is to avoid computation of column stats by engines to step on each
> other, e.g., Hive and Impala. In longer term, we may introduce a common
> representation for the column statistics stored by different engines.
> For this issue, we will add a new column 'engine' to TAB_COL_STATS HMS table
> (unpartitioned tables) and to PART_COL_STATS HMS table (partitioned tables).
> This will prevent conflicts at the column level stats.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22001) AcidUtils.getAcidState() can fail if Cleaner is removing files at the same time
[
https://issues.apache.org/jira/browse/HIVE-22001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900300#comment-16900300
]
Jason Dere commented on HIVE-22001:
---
Looks like HIVE-21225 did not include fixes for this issue after all.
> AcidUtils.getAcidState() can fail if Cleaner is removing files at the same
> time
> ---
>
> Key: HIVE-22001
> URL: https://issues.apache.org/jira/browse/HIVE-22001
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
>
> Had one user hit the following error during getSplits
> {noformat}
> 2019-07-06T14:33:03,067 ERROR [4640181a-3eb7-4b3e-9a40-d7a8de9a570c
> HiveServer2-HttpHandler-Pool: Thread-415519]: SessionState
> (SessionState.java:printError(1247)) - Vertex failed, vertexName=Map 1,
> vertexId=vertex_1560947172646_2452_6199_00, diagnostics=[Vertex
> vertex_1560947172646_2452_6199_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: hive_table initializer failed,
> vertex=vertex_1560947172646_2452_6199_00 [Map 1], java.lang.RuntimeException:
> ORC split generation failed with exception: java.io.FileNotFoundException:
> File hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070
> does not exist.
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1870)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1958)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1809)
> ... 17 more
> Caused by: java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1126)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1953)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$MetaDataFile.chooseFile(AcidUtils.java:1903)
> at
> org.apache.hadoop.hive.
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[
https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900301#comment-16900301
]
Hive QA commented on HIVE-20801:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 6 new + 562 unchanged - 0
fixed = 568 total (was 562) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18259/dev-support/hive-personality.sh
|
| git revision | master / 212b428 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18259/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18259/yetus/whitespace-eol.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18259/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Allow DbTxnManager to ignore non-ACID table locking
> -
>
> Key: HIVE-20801
> URL: https://issues.apache.org/jira/browse/HIVE-20801
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Labels: Branch3Candidate, TODOC
> Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch,
> HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch
>
>
> Enabling ACIDv1 on a cluster produces a central locking bottleneck for all
> table types, which is not always the intention.
> The Hive locking for non-acid tables are advisory (i.e a client can
> write/read without locking), which means that the implementation does not
> offer str
[jira] [Updated] (HIVE-22001) AcidUtils.getAcidState() can fail if Cleaner is removing files at the same time
[
https://issues.apache.org/jira/browse/HIVE-22001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-22001:
--
Status: Patch Available (was: Open)
> AcidUtils.getAcidState() can fail if Cleaner is removing files at the same
> time
> ---
>
> Key: HIVE-22001
> URL: https://issues.apache.org/jira/browse/HIVE-22001
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22001.1.patch
>
>
> Had one user hit the following error during getSplits
> {noformat}
> 2019-07-06T14:33:03,067 ERROR [4640181a-3eb7-4b3e-9a40-d7a8de9a570c
> HiveServer2-HttpHandler-Pool: Thread-415519]: SessionState
> (SessionState.java:printError(1247)) - Vertex failed, vertexName=Map 1,
> vertexId=vertex_1560947172646_2452_6199_00, diagnostics=[Vertex
> vertex_1560947172646_2452_6199_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: hive_table initializer failed,
> vertex=vertex_1560947172646_2452_6199_00 [Map 1], java.lang.RuntimeException:
> ORC split generation failed with exception: java.io.FileNotFoundException:
> File hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070
> does not exist.
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1870)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1958)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1809)
> ... 17 more
> Caused by: java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1126)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1953)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$MetaDataFile.chooseFile(AcidUtils.java:1903)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$MetaDataFile.isRawFormat(Ac
[jira] [Updated] (HIVE-22001) AcidUtils.getAcidState() can fail if Cleaner is removing files at the same time
[
https://issues.apache.org/jira/browse/HIVE-22001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-22001:
--
Attachment: HIVE-22001.1.patch
> AcidUtils.getAcidState() can fail if Cleaner is removing files at the same
> time
> ---
>
> Key: HIVE-22001
> URL: https://issues.apache.org/jira/browse/HIVE-22001
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22001.1.patch
>
>
> Had one user hit the following error during getSplits
> {noformat}
> 2019-07-06T14:33:03,067 ERROR [4640181a-3eb7-4b3e-9a40-d7a8de9a570c
> HiveServer2-HttpHandler-Pool: Thread-415519]: SessionState
> (SessionState.java:printError(1247)) - Vertex failed, vertexName=Map 1,
> vertexId=vertex_1560947172646_2452_6199_00, diagnostics=[Vertex
> vertex_1560947172646_2452_6199_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: hive_table initializer failed,
> vertex=vertex_1560947172646_2452_6199_00 [Map 1], java.lang.RuntimeException:
> ORC split generation failed with exception: java.io.FileNotFoundException:
> File hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070
> does not exist.
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1870)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1958)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1809)
> ... 17 more
> Caused by: java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1126)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1953)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$MetaDataFile.chooseFile(AcidUtils.java:1903)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$MetaDataFile.isRawFormat(AcidUtil
[jira] [Commented] (HIVE-22001) AcidUtils.getAcidState() can fail if Cleaner is removing files at the same time
[
https://issues.apache.org/jira/browse/HIVE-22001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900313#comment-16900313
]
Jason Dere commented on HIVE-22001:
---
[~vgumashta] [~ashutoshc] can you review?
> AcidUtils.getAcidState() can fail if Cleaner is removing files at the same
> time
> ---
>
> Key: HIVE-22001
> URL: https://issues.apache.org/jira/browse/HIVE-22001
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22001.1.patch
>
>
> Had one user hit the following error during getSplits
> {noformat}
> 2019-07-06T14:33:03,067 ERROR [4640181a-3eb7-4b3e-9a40-d7a8de9a570c
> HiveServer2-HttpHandler-Pool: Thread-415519]: SessionState
> (SessionState.java:printError(1247)) - Vertex failed, vertexName=Map 1,
> vertexId=vertex_1560947172646_2452_6199_00, diagnostics=[Vertex
> vertex_1560947172646_2452_6199_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: hive_table initializer failed,
> vertex=vertex_1560947172646_2452_6199_00 [Map 1], java.lang.RuntimeException:
> ORC split generation failed with exception: java.io.FileNotFoundException:
> File hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070
> does not exist.
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1870)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1958)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1809)
> ... 17 more
> Caused by: java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1126)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1953)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$MetaDataFile.chooseFile(AcidUtils.java:1903)
> at
> org.apa
[jira] [Work logged] (HIVE-22046) Differentiate among column stats computed by different engines
[
https://issues.apache.org/jira/browse/HIVE-22046?focusedWorklogId=289145&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289145
]
ASF GitHub Bot logged work on HIVE-22046:
-
Author: ASF GitHub Bot
Created on: 05/Aug/19 19:12
Start Date: 05/Aug/19 19:12
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #741: HIVE-22046
URL: https://github.com/apache/hive/pull/741#discussion_r310751284
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -3089,7 +3089,7 @@ public Table get_table(final String dbname, final String
name) throws MetaExcept
NoSuchObjectException {
String[] parsedDbName = parseDbName(dbname, conf);
return getTableInternal(
-parsedDbName[CAT_NAME], parsedDbName[DB_NAME], name, null, null,
false);
+parsedDbName[CAT_NAME], parsedDbName[DB_NAME], name, null, null,
false, null);
Review comment:
Started making this change and there are already some boolean in the
metastore thrift API that probably we would need to deprecate. To not make this
patch any more complex, I just documented the API properly and added a
precondition at the start of the method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 289145)
Time Spent: 1h (was: 50m)
> Differentiate among column stats computed by different engines
> --
>
> Key: HIVE-22046
> URL: https://issues.apache.org/jira/browse/HIVE-22046
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22046.01.patch, HIVE-22046.02.patch,
> HIVE-22046.03.patch, HIVE-22046.04.patch, HIVE-22046.05.patch,
> HIVE-22046.06.patch, HIVE-22046.07.patch, HIVE-22046.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The goal is to avoid computation of column stats by engines to step on each
> other, e.g., Hive and Impala. In longer term, we may introduce a common
> representation for the column statistics stored by different engines.
> For this issue, we will add a new column 'engine' to TAB_COL_STATS HMS table
> (unpartitioned tables) and to PART_COL_STATS HMS table (partitioned tables).
> This will prevent conflicts at the column level stats.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (HIVE-22046) Differentiate among column stats computed by different engines
[
https://issues.apache.org/jira/browse/HIVE-22046?focusedWorklogId=289146&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289146
]
ASF GitHub Bot logged work on HIVE-22046:
-
Author: ASF GitHub Bot
Created on: 05/Aug/19 19:12
Start Date: 05/Aug/19 19:12
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #741: HIVE-22046
URL: https://github.com/apache/hive/pull/741#discussion_r310751356
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -3255,14 +3255,14 @@ public Table get_table_core(
final String name,
final String writeIdList)
throws MetaException, NoSuchObjectException {
- return get_table_core(catName, dbname, name, writeIdList, false);
+ return get_table_core(catName, dbname, name, writeIdList, false, null);
Review comment:
Same as above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 289146)
Time Spent: 1h 10m (was: 1h)
> Differentiate among column stats computed by different engines
> --
>
> Key: HIVE-22046
> URL: https://issues.apache.org/jira/browse/HIVE-22046
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22046.01.patch, HIVE-22046.02.patch,
> HIVE-22046.03.patch, HIVE-22046.04.patch, HIVE-22046.05.patch,
> HIVE-22046.06.patch, HIVE-22046.07.patch, HIVE-22046.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The goal is to avoid computation of column stats by engines to step on each
> other, e.g., Hive and Impala. In longer term, we may introduce a common
> representation for the column statistics stored by different engines.
> For this issue, we will add a new column 'engine' to TAB_COL_STATS HMS table
> (unpartitioned tables) and to PART_COL_STATS HMS table (partitioned tables).
> This will prevent conflicts at the column level stats.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22046) Differentiate among column stats computed by different engines
[ https://issues.apache.org/jira/browse/HIVE-22046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22046: --- Attachment: HIVE-22046.08.patch > Differentiate among column stats computed by different engines > -- > > Key: HIVE-22046 > URL: https://issues.apache.org/jira/browse/HIVE-22046 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22046.01.patch, HIVE-22046.02.patch, > HIVE-22046.03.patch, HIVE-22046.04.patch, HIVE-22046.05.patch, > HIVE-22046.06.patch, HIVE-22046.07.patch, HIVE-22046.08.patch, > HIVE-22046.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The goal is to avoid computation of column stats by engines to step on each > other, e.g., Hive and Impala. In longer term, we may introduce a common > representation for the column statistics stored by different engines. > For this issue, we will add a new column 'engine' to TAB_COL_STATS HMS table > (unpartitioned tables) and to PART_COL_STATS HMS table (partitioned tables). > This will prevent conflicts at the column level stats. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[
https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900338#comment-16900338
]
Hive QA commented on HIVE-20801:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976705/HIVE-20801.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16724 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18259/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18259/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18259/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976705 - PreCommit-HIVE-Build
> ACID: Allow DbTxnManager to ignore non-ACID table locking
> -
>
> Key: HIVE-20801
> URL: https://issues.apache.org/jira/browse/HIVE-20801
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Labels: Branch3Candidate, TODOC
> Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch,
> HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch
>
>
> Enabling ACIDv1 on a cluster produces a central locking bottleneck for all
> table types, which is not always the intention.
> The Hive locking for non-acid tables are advisory (i.e a client can
> write/read without locking), which means that the implementation does not
> offer strong consistency despite the lock manager consuming resources
> centrally.
> Disabling this lock acquisition would improve the performance of non-ACID
> tables co-existing with a globally configured DbTxnManager implementation.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22074) Slow compilation due to IN to OR transformation
[ https://issues.apache.org/jira/browse/HIVE-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22074: --- Attachment: HIVE-22074.3.patch > Slow compilation due to IN to OR transformation > --- > > Key: HIVE-22074 > URL: https://issues.apache.org/jira/browse/HIVE-22074 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch, > HIVE-22074.3.patch > > > Currently Hive transform IN expressions to OR to apply various CBO rules. > This incur significant performance hit if IN consist of large number of > expressions. > It is better to not transform IN expressions to OR in such cases because > overall benefit of various optimizations/transformations is unrealized due to > the compilation overhead -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22074) Slow compilation due to IN to OR transformation
[ https://issues.apache.org/jira/browse/HIVE-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22074: --- Status: Patch Available (was: Open) > Slow compilation due to IN to OR transformation > --- > > Key: HIVE-22074 > URL: https://issues.apache.org/jira/browse/HIVE-22074 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch, > HIVE-22074.3.patch > > > Currently Hive transform IN expressions to OR to apply various CBO rules. > This incur significant performance hit if IN consist of large number of > expressions. > It is better to not transform IN expressions to OR in such cases because > overall benefit of various optimizations/transformations is unrealized due to > the compilation overhead -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22074) Slow compilation due to IN to OR transformation
[ https://issues.apache.org/jira/browse/HIVE-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22074: --- Status: Open (was: Patch Available) > Slow compilation due to IN to OR transformation > --- > > Key: HIVE-22074 > URL: https://issues.apache.org/jira/browse/HIVE-22074 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch, > HIVE-22074.3.patch > > > Currently Hive transform IN expressions to OR to apply various CBO rules. > This incur significant performance hit if IN consist of large number of > expressions. > It is better to not transform IN expressions to OR in such cases because > overall benefit of various optimizations/transformations is unrealized due to > the compilation overhead -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22046) Differentiate among column stats computed by different engines
[ https://issues.apache.org/jira/browse/HIVE-22046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900376#comment-16900376 ] Jason Dere commented on HIVE-22046: --- +1 > Differentiate among column stats computed by different engines > -- > > Key: HIVE-22046 > URL: https://issues.apache.org/jira/browse/HIVE-22046 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22046.01.patch, HIVE-22046.02.patch, > HIVE-22046.03.patch, HIVE-22046.04.patch, HIVE-22046.05.patch, > HIVE-22046.06.patch, HIVE-22046.07.patch, HIVE-22046.08.patch, > HIVE-22046.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The goal is to avoid computation of column stats by engines to step on each > other, e.g., Hive and Impala. In longer term, we may introduce a common > representation for the column statistics stored by different engines. > For this issue, we will add a new column 'engine' to TAB_COL_STATS HMS table > (unpartitioned tables) and to PART_COL_STATS HMS table (partitioned tables). > This will prevent conflicts at the column level stats. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21241:
--
Status: Patch Available (was: Open)
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch,
> HIVE-21241.3.patch, HIVE-21241.4.patch, HIVE-21241.5.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21241:
--
Status: Patch Available (was: Open)
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch,
> HIVE-21241.3.patch, HIVE-21241.4.patch, HIVE-21241.5.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21241:
--
Status: Open (was: Patch Available)
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch,
> HIVE-21241.3.patch, HIVE-21241.4.patch, HIVE-21241.5.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21241:
--
Attachment: HIVE-21241.5.patch
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch,
> HIVE-21241.3.patch, HIVE-21241.4.patch, HIVE-21241.5.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21241:
--
Status: Open (was: Patch Available)
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch,
> HIVE-21241.3.patch, HIVE-21241.4.patch, HIVE-21241.5.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-6589) Automatically add partitions for external tables
[
https://issues.apache.org/jira/browse/HIVE-6589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900386#comment-16900386
]
Amogh Antarkar commented on HIVE-6589:
--
Please let me know if there are any updates on this issue or any workarounds to
it. Since Flume is used to stream into hdfs, it would be great if we do not
have to run a separate batch job to detect partitions. Any help appreciated!
> Automatically add partitions for external tables
>
>
> Key: HIVE-6589
> URL: https://issues.apache.org/jira/browse/HIVE-6589
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 0.14.0
>Reporter: Ken Dallmeyer
>Priority: Major
>
> I have a data stream being loaded into Hadoop via Flume. It loads into a date
> partition folder in HDFS. The path looks like this:
> {code}/flume/my_data//MM/DD/HH
> /flume/my_data/2014/03/02/01
> /flume/my_data/2014/03/02/02
> /flume/my_data/2014/03/02/03{code}
> On top of it I create an EXTERNAL hive table to do querying. As of now, I
> have to manually add partitions. What I want is for EXTERNAL tables, Hive
> should "discover" those partitions. Additionally I would like to specify a
> partition pattern so that when I query Hive will know to use the partition
> pattern to find the HDFS folder.
> So something like this:
> {code}CREATE EXTERNAL TABLE my_data (
> col1 STRING,
> col2 INT
> )
> PARTITIONED BY (
> dt STRING,
> hour STRING
> )
> LOCATION
> '/flume/mydata'
> TBLPROPERTIES (
> 'hive.partition.spec' = 'dt=$Y-$M-$D, hour=$H',
> 'hive.partition.spec.location' = '$Y/$M/$D/$H',
> );
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (HIVE-6589) Automatically add partitions for external tables
[
https://issues.apache.org/jira/browse/HIVE-6589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900386#comment-16900386
]
Amogh Antarkar edited comment on HIVE-6589 at 8/5/19 8:12 PM:
--
Please let me know if there are any updates on this issue or any workarounds to
it. Since Flume is used to stream into hdfs, it would be great if we do not
have to run a separate batch job to detect partitions for hive table. Any help
appreciated!
was (Author: amoghantarkar):
Please let me know if there are any updates on this issue or any workarounds to
it. Since Flume is used to stream into hdfs, it would be great if we do not
have to run a separate batch job to detect partitions. Any help appreciated!
> Automatically add partitions for external tables
>
>
> Key: HIVE-6589
> URL: https://issues.apache.org/jira/browse/HIVE-6589
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 0.14.0
>Reporter: Ken Dallmeyer
>Priority: Major
>
> I have a data stream being loaded into Hadoop via Flume. It loads into a date
> partition folder in HDFS. The path looks like this:
> {code}/flume/my_data//MM/DD/HH
> /flume/my_data/2014/03/02/01
> /flume/my_data/2014/03/02/02
> /flume/my_data/2014/03/02/03{code}
> On top of it I create an EXTERNAL hive table to do querying. As of now, I
> have to manually add partitions. What I want is for EXTERNAL tables, Hive
> should "discover" those partitions. Additionally I would like to specify a
> partition pattern so that when I query Hive will know to use the partition
> pattern to find the HDFS folder.
> So something like this:
> {code}CREATE EXTERNAL TABLE my_data (
> col1 STRING,
> col2 INT
> )
> PARTITIONED BY (
> dt STRING,
> hour STRING
> )
> LOCATION
> '/flume/mydata'
> TBLPROPERTIES (
> 'hive.partition.spec' = 'dt=$Y-$M-$D, hour=$H',
> 'hive.partition.spec.location' = '$Y/$M/$D/$H',
> );
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (HIVE-6589) Automatically add partitions for external tables
[
https://issues.apache.org/jira/browse/HIVE-6589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900386#comment-16900386
]
Amogh Antarkar edited comment on HIVE-6589 at 8/5/19 8:13 PM:
--
Please let us know if there are any updates on this issue or any workarounds to
it. Since Flume is used to stream into hdfs, it would be great if we do not
have to run a separate batch job to detect partitions for hive table. Any help
appreciated!
was (Author: amoghantarkar):
Please let me know if there are any updates on this issue or any workarounds to
it. Since Flume is used to stream into hdfs, it would be great if we do not
have to run a separate batch job to detect partitions for hive table. Any help
appreciated!
> Automatically add partitions for external tables
>
>
> Key: HIVE-6589
> URL: https://issues.apache.org/jira/browse/HIVE-6589
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 0.14.0
>Reporter: Ken Dallmeyer
>Priority: Major
>
> I have a data stream being loaded into Hadoop via Flume. It loads into a date
> partition folder in HDFS. The path looks like this:
> {code}/flume/my_data//MM/DD/HH
> /flume/my_data/2014/03/02/01
> /flume/my_data/2014/03/02/02
> /flume/my_data/2014/03/02/03{code}
> On top of it I create an EXTERNAL hive table to do querying. As of now, I
> have to manually add partitions. What I want is for EXTERNAL tables, Hive
> should "discover" those partitions. Additionally I would like to specify a
> partition pattern so that when I query Hive will know to use the partition
> pattern to find the HDFS folder.
> So something like this:
> {code}CREATE EXTERNAL TABLE my_data (
> col1 STRING,
> col2 INT
> )
> PARTITIONED BY (
> dt STRING,
> hour STRING
> )
> LOCATION
> '/flume/mydata'
> TBLPROPERTIES (
> 'hive.partition.spec' = 'dt=$Y-$M-$D, hour=$H',
> 'hive.partition.spec.location' = '$Y/$M/$D/$H',
> );
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22080) Prevent implicit conversion from String/char/varchar to double/decimal
[ https://issues.apache.org/jira/browse/HIVE-22080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22080: Status: Open (was: Patch Available) Schema Evolution tests failing fixed in the next patch. > Prevent implicit conversion from String/char/varchar to double/decimal > -- > > Key: HIVE-22080 > URL: https://issues.apache.org/jira/browse/HIVE-22080 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22080.1.patch > > > Implicit conversion from String family types to any non-string family types > are invalid. User can force the conversion by turning off the setting > hive.metastore.disallow.incompatible.col.type.changes. If not turned off, > such a conversion should throw error. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22080) Prevent implicit conversion from String/char/varchar to double/decimal
[ https://issues.apache.org/jira/browse/HIVE-22080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22080: Attachment: HIVE-22080.2.patch Status: Patch Available (was: Open) > Prevent implicit conversion from String/char/varchar to double/decimal > -- > > Key: HIVE-22080 > URL: https://issues.apache.org/jira/browse/HIVE-22080 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22080.1.patch, HIVE-22080.2.patch > > > Implicit conversion from String family types to any non-string family types > are invalid. User can force the conversion by turning off the setting > hive.metastore.disallow.incompatible.col.type.changes. If not turned off, > such a conversion should throw error. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21828) Tez: Use a pre-parsed TezConfiguration from DagUtils
[
https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900422#comment-16900422
]
Hive QA commented on HIVE-21828:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18260/dev-support/hive-personality.sh
|
| git revision | master / 212b428 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18260/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Tez: Use a pre-parsed TezConfiguration from DagUtils
>
>
> Key: HIVE-21828
> URL: https://issues.apache.org/jira/browse/HIVE-21828
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21828.1.patch, HIVE-21828.2.patch,
> HIVE-21828.5.patch, HIVE-21828.6.patch, HIVE-21828.7.patch, HIVE-21828.8.patch
>
>
> The HS2 tez-site.xml does not change dynamically - the XML parsed components
> of the config can be obtained statically and kept across sessions.
> This allows for the replacing of "new TezConfiguration()" with a HS2 local
> version instead.
> The configuration object however has to reference the right resource file
> (i.e location of tez-site.xml) without reparsing it for each query.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21828) Tez: Use a pre-parsed TezConfiguration from DagUtils
[
https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900443#comment-16900443
]
Hive QA commented on HIVE-21828:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976706/HIVE-21828.8.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16723 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18260/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18260/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18260/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976706 - PreCommit-HIVE-Build
> Tez: Use a pre-parsed TezConfiguration from DagUtils
>
>
> Key: HIVE-21828
> URL: https://issues.apache.org/jira/browse/HIVE-21828
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21828.1.patch, HIVE-21828.2.patch,
> HIVE-21828.5.patch, HIVE-21828.6.patch, HIVE-21828.7.patch, HIVE-21828.8.patch
>
>
> The HS2 tez-site.xml does not change dynamically - the XML parsed components
> of the config can be obtained statically and kept across sessions.
> This allows for the replacing of "new TezConfiguration()" with a HS2 local
> version instead.
> The configuration object however has to reference the right resource file
> (i.e location of tez-site.xml) without reparsing it for each query.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22081) Hivemetastore Performance: Compaction Initiator Thread overwhelmed if there are too many Table/partitions are eligible for compaction
[ https://issues.apache.org/jira/browse/HIVE-22081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22081: -- Status: Open (was: Patch Available) > Hivemetastore Performance: Compaction Initiator Thread overwhelmed if there > are too many Table/partitions are eligible for compaction > -- > > Key: HIVE-22081 > URL: https://issues.apache.org/jira/browse/HIVE-22081 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22081.patch > > > if Automatic Compaction is turned on, Initiator thread check for potential > table/partitions which are eligible for compactions and run some checks in > for loop before requesting compaction for eligibles. Though initiator thread > is configured to run at interval 5 min default, in case of many objects it > keeps on running as these checks are IO intensive and hog cpu. > In the proposed changes, I am planning to do > 1. passing less object to for loop by filtering out the objects based on the > condition which we are checking within the loop. > 2. Doing Async call using future to determine compaction type(this is where > we do FileSystem calls) -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22081) Hivemetastore Performance: Compaction Initiator Thread overwhelmed if there are too many Table/partitions are eligible for compaction
[ https://issues.apache.org/jira/browse/HIVE-22081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22081: -- Attachment: HIVE-21917.01.patch Status: Patch Available (was: Open) > Hivemetastore Performance: Compaction Initiator Thread overwhelmed if there > are too many Table/partitions are eligible for compaction > -- > > Key: HIVE-22081 > URL: https://issues.apache.org/jira/browse/HIVE-22081 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21917.01.patch, HIVE-22081.patch > > > if Automatic Compaction is turned on, Initiator thread check for potential > table/partitions which are eligible for compactions and run some checks in > for loop before requesting compaction for eligibles. Though initiator thread > is configured to run at interval 5 min default, in case of many objects it > keeps on running as these checks are IO intensive and hog cpu. > In the proposed changes, I am planning to do > 1. passing less object to for loop by filtering out the objects based on the > condition which we are checking within the loop. > 2. Doing Async call using future to determine compaction type(this is where > we do FileSystem calls) -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900483#comment-16900483
]
Hive QA commented on HIVE-4605:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18261/dev-support/hive-personality.sh
|
| git revision | master / 212b428 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18261/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.3.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900504#comment-16900504
]
Hive QA commented on HIVE-4605:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976723/HIVE-4605.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 16723 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=157)
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=360)
org.apache.hadoop.hive.metastore.TestMetaStoreAuthorization.testMetaStoreAuthorization
(batchId=242)
org.apache.hadoop.hive.ql.TestTxnExIm.testExportDefaultDb (batchId=331)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18261/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18261/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18261/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976723 - PreCommit-HIVE-Build
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.3.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> ... 7 more
> {code}
--
This me
[jira] [Commented] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900511#comment-16900511
]
Hive QA commented on HIVE-22079:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 8 new + 16 unchanged - 0 fixed
= 24 total (was 16) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18262/dev-support/hive-personality.sh
|
| git revision | master / 212b428 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18262/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18262/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21828) Tez: Use a pre-parsed TezConfiguration from DagUtils
[ https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900524#comment-16900524 ] Gopal V commented on HIVE-21828: LGTM - +1 > Tez: Use a pre-parsed TezConfiguration from DagUtils > > > Key: HIVE-21828 > URL: https://issues.apache.org/jira/browse/HIVE-21828 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21828.1.patch, HIVE-21828.2.patch, > HIVE-21828.5.patch, HIVE-21828.6.patch, HIVE-21828.7.patch, HIVE-21828.8.patch > > > The HS2 tez-site.xml does not change dynamically - the XML parsed components > of the config can be obtained statically and kept across sessions. > This allows for the replacing of "new TezConfiguration()" with a HS2 local > version instead. > The configuration object however has to reference the right resource file > (i.e location of tez-site.xml) without reparsing it for each query. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900539#comment-16900539
]
Hive QA commented on HIVE-22079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976735/HIVE-22079.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 16723 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_join0]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_unqual1]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_unqual2]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_unqual4]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonmr_fetch] (batchId=22)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_gby]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_join]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_gby]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_join]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_semijoin]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_semijoin]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_assertion_type]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=193)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_gby]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_limit]
(batchId=153)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_semijoin]
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_cond_pushdown_unqual1]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_cond_pushdown_unqual2]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_cond_pushdown_unqual4]
(batchId=113)
org.apache.hive.service.server.TestInformationSchemaWithPrivilege.test
(batchId=280)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18262/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18262/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18262/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976735 - PreCommit-HIVE-Build
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-16587) NPE when inserting complex types with nested null values
[
https://issues.apache.org/jira/browse/HIVE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900548#comment-16900548
]
Hive QA commented on HIVE-16587:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18263/dev-support/hive-personality.sh
|
| git revision | master / 212b428 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18263/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NPE when inserting complex types with nested null values
>
>
> Key: HIVE-16587
> URL: https://issues.apache.org/jira/browse/HIVE-16587
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1
>Reporter: Jason Dere
>Assignee: Naresh P R
>Priority: Major
> Attachments: HIVE-16587.patch
>
>
> {noformat}
> CREATE TABLE complex1 (c0 int, c1 array, c2 map, c3
> struct>, c4 array f3:array>>)
> insert into complex1
> select 3, array(1, 2, null), map(1, 'one', 2, null), named_struct('f1',
> cast(null as int), 'f2', cast(null as string), 'f3', array(1,2,null)),
> array(named_struct('f1', 11, 'f2', 'two', 'f3', array(2,3,4)))
> {noformat}
> Gives the following error:
> {noformat}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:207)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:291)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:255)
> at
> org.apa
[jira] [Updated] (HIVE-22040) Drop partition throws exception with 'Failed to delete parent: File does not exist' when the partition's parent path does not exists
[
https://issues.apache.org/jira/browse/HIVE-22040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiepengjie updated HIVE-22040:
--
Attachment: (was: HIVE-22040.03.patch)
> Drop partition throws exception with 'Failed to delete parent: File does not
> exist' when the partition's parent path does not exists
>
>
> Key: HIVE-22040
> URL: https://issues.apache.org/jira/browse/HIVE-22040
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: xiepengjie
>Assignee: xiepengjie
>Priority: Major
> Attachments: HIVE-22040.01.patch, HIVE-22040.02.patch,
> HIVE-22040.patch
>
>
> I create a manage table with multi partition columns, when i try to drop
> partition throws exception with 'Failed to delete parent: File does not
> exist' when the partition's parent path does not exist. The partition's
> metadata in mysql has been deleted, but the exception is still thrown. it
> will fail if connecting hiveserver2 with jdbc by java, this problem also
> exists in master branch, I think it is very unfriendly and we should fix it.
> Example:
> – First, create manage table with nulti partition columns, and add partitions:
> {code:java}
> drop table if exists t1;
> create table t1 (c1 int) partitioned by (year string, month string, day
> string);
> alter table t1 add partition(year='2019', month='07', day='01');{code}
> – Second, delete the path of partition 'month=07':
> {code:java}
> hadoop fs -rm -r
> /user/hadoop/xiepengjietest.db/drop_partition/year=2019/month=07{code}
> -- Third, when i try to drop partition, the metastore throws exception with
> 'Failed to delete parent: File does not exist' .
> {code:java}
> alter table t1 drop partition(year='2019', month='07', day='01');
> {code}
> exception like this:
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Failed to delete parent: File
> does not exist:
> /user/hadoop/xiepengjietest.db/drop_partition/year=2019/month=07
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getContentSummaryInt(FSDirStatAndListingOp.java:493)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getContentSummary(FSDirStatAndListingOp.java:140)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getContentSummary(FSNamesystem.java:3995)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getContentSummary(NameNodeRpcServer.java:1202)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getContentSummary(ClientNamenodeProtocolServerSideTranslatorPB.java:883)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22040) Drop partition throws exception with 'Failed to delete parent: File does not exist' when the partition's parent path does not exists
[
https://issues.apache.org/jira/browse/HIVE-22040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiepengjie updated HIVE-22040:
--
Attachment: HIVE-22040.03.patch
> Drop partition throws exception with 'Failed to delete parent: File does not
> exist' when the partition's parent path does not exists
>
>
> Key: HIVE-22040
> URL: https://issues.apache.org/jira/browse/HIVE-22040
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: xiepengjie
>Assignee: xiepengjie
>Priority: Major
> Attachments: HIVE-22040.01.patch, HIVE-22040.02.patch,
> HIVE-22040.03.patch, HIVE-22040.patch
>
>
> I create a manage table with multi partition columns, when i try to drop
> partition throws exception with 'Failed to delete parent: File does not
> exist' when the partition's parent path does not exist. The partition's
> metadata in mysql has been deleted, but the exception is still thrown. it
> will fail if connecting hiveserver2 with jdbc by java, this problem also
> exists in master branch, I think it is very unfriendly and we should fix it.
> Example:
> – First, create manage table with nulti partition columns, and add partitions:
> {code:java}
> drop table if exists t1;
> create table t1 (c1 int) partitioned by (year string, month string, day
> string);
> alter table t1 add partition(year='2019', month='07', day='01');{code}
> – Second, delete the path of partition 'month=07':
> {code:java}
> hadoop fs -rm -r
> /user/hadoop/xiepengjietest.db/drop_partition/year=2019/month=07{code}
> -- Third, when i try to drop partition, the metastore throws exception with
> 'Failed to delete parent: File does not exist' .
> {code:java}
> alter table t1 drop partition(year='2019', month='07', day='01');
> {code}
> exception like this:
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Failed to delete parent: File
> does not exist:
> /user/hadoop/xiepengjietest.db/drop_partition/year=2019/month=07
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getContentSummaryInt(FSDirStatAndListingOp.java:493)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getContentSummary(FSDirStatAndListingOp.java:140)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getContentSummary(FSNamesystem.java:3995)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getContentSummary(NameNodeRpcServer.java:1202)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getContentSummary(ClientNamenodeProtocolServerSideTranslatorPB.java:883)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-16587) NPE when inserting complex types with nested null values
[
https://issues.apache.org/jira/browse/HIVE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900572#comment-16900572
]
Hive QA commented on HIVE-16587:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976742/HIVE-16587.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16724 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18263/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18263/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18263/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976742 - PreCommit-HIVE-Build
> NPE when inserting complex types with nested null values
>
>
> Key: HIVE-16587
> URL: https://issues.apache.org/jira/browse/HIVE-16587
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1
>Reporter: Jason Dere
>Assignee: Naresh P R
>Priority: Major
> Attachments: HIVE-16587.patch
>
>
> {noformat}
> CREATE TABLE complex1 (c0 int, c1 array, c2 map, c3
> struct>, c4 array f3:array>>)
> insert into complex1
> select 3, array(1, 2, null), map(1, 'one', 2, null), named_struct('f1',
> cast(null as int), 'f2', cast(null as string), 'f3', array(1,2,null)),
> array(named_struct('f1', 11, 'f2', 'two', 'f3', array(2,3,4)))
> {noformat}
> Gives the following error:
> {noformat}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:207)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:291)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:255)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:531)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
> at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$Proxy126.executeStatementAsync(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:530)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getWritableSize(StatsUtils.java:1144)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getSizeOfMap(StatsUtils.java:1106)
> at
>
[jira] [Commented] (HIVE-7044) ORC Vector: column of empty strings is read back as null
[ https://issues.apache.org/jira/browse/HIVE-7044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900574#comment-16900574 ] Hui An commented on HIVE-7044: -- Looks like this bug has already fixed in branch-3.1, but I do think we should add this test file to the project. > ORC Vector: column of empty strings is read back as null > > > Key: HIVE-7044 > URL: https://issues.apache.org/jira/browse/HIVE-7044 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 0.13.0 >Reporter: Dain Sundstrom >Assignee: Jitendra Nath Pandey >Priority: Blocker > Labels: orcfile, vector > Attachments: TestOrcEmptyString.java > > > If I write a column of empty string values, the vectorized read code returns > a vector of nulls, but the non-vectorized code returns the correct values. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22070) metadata discovery shows no tables if database name has underscore (_)
[
https://issues.apache.org/jira/browse/HIVE-22070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900584#comment-16900584
]
Rajkumar Singh commented on HIVE-22070:
---
This is fixed via https://issues.apache.org/jira/browse/HIVE-20568, can you
please try the patch and let me know the results?
> metadata discovery shows no tables if database name has underscore (_)
> --
>
> Key: HIVE-22070
> URL: https://issues.apache.org/jira/browse/HIVE-22070
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 3.0.0
> Environment: HDP 3.1, Denodo 7, Informatica EDC 10.2.2,
> Ranger authentication for Hive enabled.
>Reporter: Luo Donghua
>Priority: Major
>
> We discovered follow behaviors:
> connect to hive database using beeline is okay, and able to list table using
> "show tables" command
> when we connect to JDBC drivers from Denodo or informatica EDC, we can't
> browse the table list if database name has underscore ("_"), for example
> abc_mart
> after create new database with name like abcmart, it works fine for
> denodo/informatica EDC
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22046) Differentiate among column stats computed by different engines
[ https://issues.apache.org/jira/browse/HIVE-22046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22046: --- Attachment: HIVE-22046.09.patch > Differentiate among column stats computed by different engines > -- > > Key: HIVE-22046 > URL: https://issues.apache.org/jira/browse/HIVE-22046 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22046.01.patch, HIVE-22046.02.patch, > HIVE-22046.03.patch, HIVE-22046.04.patch, HIVE-22046.05.patch, > HIVE-22046.06.patch, HIVE-22046.07.patch, HIVE-22046.08.patch, > HIVE-22046.09.patch, HIVE-22046.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The goal is to avoid computation of column stats by engines to step on each > other, e.g., Hive and Impala. In longer term, we may introduce a common > representation for the column statistics stored by different engines. > For this issue, we will add a new column 'engine' to TAB_COL_STATS HMS table > (unpartitioned tables) and to PART_COL_STATS HMS table (partitioned tables). > This will prevent conflicts at the column level stats. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22001) AcidUtils.getAcidState() can fail if Cleaner is removing files at the same time
[
https://issues.apache.org/jira/browse/HIVE-22001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900591#comment-16900591
]
Hive QA commented on HIVE-22001:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18264/dev-support/hive-personality.sh
|
| git revision | master / 212b428 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18264/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> AcidUtils.getAcidState() can fail if Cleaner is removing files at the same
> time
> ---
>
> Key: HIVE-22001
> URL: https://issues.apache.org/jira/browse/HIVE-22001
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22001.1.patch
>
>
> Had one user hit the following error during getSplits
> {noformat}
> 2019-07-06T14:33:03,067 ERROR [4640181a-3eb7-4b3e-9a40-d7a8de9a570c
> HiveServer2-HttpHandler-Pool: Thread-415519]: SessionState
> (SessionState.java:printError(1247)) - Vertex failed, vertexName=Map 1,
> vertexId=vertex_1560947172646_2452_6199_00, diagnostics=[Vertex
> vertex_1560947172646_2452_6199_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: hive_table initializer failed,
> vertex=vertex_1560947172646_2452_6199_00 [Map 1], java.lang.RuntimeException:
> ORC split generation failed with exception: java.io.FileNotFoundException:
> File hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070
> does not exist.
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1870)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1958)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779)
> at
> org.apache.hadoop.hive.ql.exec.
[jira] [Commented] (HIVE-22001) AcidUtils.getAcidState() can fail if Cleaner is removing files at the same time
[
https://issues.apache.org/jira/browse/HIVE-22001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900608#comment-16900608
]
Hive QA commented on HIVE-22001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976744/HIVE-22001.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16723 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18264/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18264/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18264/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976744 - PreCommit-HIVE-Build
> AcidUtils.getAcidState() can fail if Cleaner is removing files at the same
> time
> ---
>
> Key: HIVE-22001
> URL: https://issues.apache.org/jira/browse/HIVE-22001
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22001.1.patch
>
>
> Had one user hit the following error during getSplits
> {noformat}
> 2019-07-06T14:33:03,067 ERROR [4640181a-3eb7-4b3e-9a40-d7a8de9a570c
> HiveServer2-HttpHandler-Pool: Thread-415519]: SessionState
> (SessionState.java:printError(1247)) - Vertex failed, vertexName=Map 1,
> vertexId=vertex_1560947172646_2452_6199_00, diagnostics=[Vertex
> vertex_1560947172646_2452_6199_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: hive_table initializer failed,
> vertex=vertex_1560947172646_2452_6199_00 [Map 1], java.lang.RuntimeException:
> ORC split generation failed with exception: java.io.FileNotFoundException:
> File hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070
> does not exist.
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1870)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1958)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1809)
> ... 17 more
> Caused by: java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.l
[jira] [Commented] (HIVE-22001) AcidUtils.getAcidState() can fail if Cleaner is removing files at the same time
[
https://issues.apache.org/jira/browse/HIVE-22001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900611#comment-16900611
]
Ashutosh Chauhan commented on HIVE-22001:
-
+1 during commit can you please also add a comment on when exactly files may
get deleted and why this is OK for correctness?
> AcidUtils.getAcidState() can fail if Cleaner is removing files at the same
> time
> ---
>
> Key: HIVE-22001
> URL: https://issues.apache.org/jira/browse/HIVE-22001
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22001.1.patch
>
>
> Had one user hit the following error during getSplits
> {noformat}
> 2019-07-06T14:33:03,067 ERROR [4640181a-3eb7-4b3e-9a40-d7a8de9a570c
> HiveServer2-HttpHandler-Pool: Thread-415519]: SessionState
> (SessionState.java:printError(1247)) - Vertex failed, vertexName=Map 1,
> vertexId=vertex_1560947172646_2452_6199_00, diagnostics=[Vertex
> vertex_1560947172646_2452_6199_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: hive_table initializer failed,
> vertex=vertex_1560947172646_2452_6199_00 [Map 1], java.lang.RuntimeException:
> ORC split generation failed with exception: java.io.FileNotFoundException:
> File hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070
> does not exist.
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1870)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1958)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1809)
> ... 17 more
> Caused by: java.io.FileNotFoundException: File
> hdfs://path/to/hive_table/oiddatemmdd=20190706/delta_0987070_0987070 does
> not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1126)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1953)
> at
> org.apache.
[jira] [Updated] (HIVE-21828) Tez: Use a pre-parsed TezConfiguration from DagUtils
[ https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-21828: Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Attila! > Tez: Use a pre-parsed TezConfiguration from DagUtils > > > Key: HIVE-21828 > URL: https://issues.apache.org/jira/browse/HIVE-21828 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21828.1.patch, HIVE-21828.2.patch, > HIVE-21828.5.patch, HIVE-21828.6.patch, HIVE-21828.7.patch, HIVE-21828.8.patch > > > The HS2 tez-site.xml does not change dynamically - the XML parsed components > of the config can be obtained statically and kept across sessions. > This allows for the replacing of "new TezConfiguration()" with a HS2 local > version instead. > The configuration object however has to reference the right resource file > (i.e location of tez-site.xml) without reparsing it for each query. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22046) Differentiate among column stats computed by different engines
[
https://issues.apache.org/jira/browse/HIVE-22046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900651#comment-16900651
]
Hive QA commented on HIVE-22046:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
32s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
18s{color} | {color:blue} standalone-metastore/metastore-server in master has
180 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
48s{color} | {color:blue} itests/util in master has 44 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 8 new + 402 unchanged - 6 fixed = 410 total (was 408) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 76 new + 1941 unchanged - 38 fixed = 2017 total (was 1979) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 5 new + 1126 unchanged - 3
fixed = 1131 total (was 1129) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} itests/hcatalog-unit: The patch generated 3 new + 26
unchanged - 1 fixed = 29 total (was 27) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 13 new + 307
unchanged - 1 fixed = 320 total (was 308) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
23s{color} | {color:red} standalone-metastore/metastore-server generated 2 new
+ 178 unchanged - 2 fixed = 180 total (was 180) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
20s{color} | {color:red} standalone-metastore_metastore-common generated 2 new
+ 49 unchanged - 0 fixed = 51 total (was 49) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | instanceof will always return false in
org.
[jira] [Commented] (HIVE-22046) Differentiate among column stats computed by different engines
[
https://issues.apache.org/jira/browse/HIVE-22046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900659#comment-16900659
]
Hive QA commented on HIVE-22046:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976781/HIVE-22046.09.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16723 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18265/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18265/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18265/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976781 - PreCommit-HIVE-Build
> Differentiate among column stats computed by different engines
> --
>
> Key: HIVE-22046
> URL: https://issues.apache.org/jira/browse/HIVE-22046
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22046.01.patch, HIVE-22046.02.patch,
> HIVE-22046.03.patch, HIVE-22046.04.patch, HIVE-22046.05.patch,
> HIVE-22046.06.patch, HIVE-22046.07.patch, HIVE-22046.08.patch,
> HIVE-22046.09.patch, HIVE-22046.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The goal is to avoid computation of column stats by engines to step on each
> other, e.g., Hive and Impala. In longer term, we may introduce a common
> representation for the column statistics stored by different engines.
> For this issue, we will add a new column 'engine' to TAB_COL_STATS HMS table
> (unpartitioned tables) and to PART_COL_STATS HMS table (partitioned tables).
> This will prevent conflicts at the column level stats.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22046) Differentiate among column stats computed by different engines
[ https://issues.apache.org/jira/browse/HIVE-22046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22046: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master, thanks [~jdere]. > Differentiate among column stats computed by different engines > -- > > Key: HIVE-22046 > URL: https://issues.apache.org/jira/browse/HIVE-22046 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22046.01.patch, HIVE-22046.02.patch, > HIVE-22046.03.patch, HIVE-22046.04.patch, HIVE-22046.05.patch, > HIVE-22046.06.patch, HIVE-22046.07.patch, HIVE-22046.08.patch, > HIVE-22046.09.patch, HIVE-22046.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The goal is to avoid computation of column stats by engines to step on each > other, e.g., Hive and Impala. In longer term, we may introduce a common > representation for the column statistics stored by different engines. > For this issue, we will add a new column 'engine' to TAB_COL_STATS HMS table > (unpartitioned tables) and to PART_COL_STATS HMS table (partitioned tables). > This will prevent conflicts at the column level stats. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (HIVE-22046) Differentiate among column stats computed by different engines
[ https://issues.apache.org/jira/browse/HIVE-22046?focusedWorklogId=289453&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289453 ] ASF GitHub Bot logged work on HIVE-22046: - Author: ASF GitHub Bot Created on: 06/Aug/19 06:12 Start Date: 06/Aug/19 06:12 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #741: HIVE-22046 URL: https://github.com/apache/hive/pull/741 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289453) Time Spent: 1h 20m (was: 1h 10m) > Differentiate among column stats computed by different engines > -- > > Key: HIVE-22046 > URL: https://issues.apache.org/jira/browse/HIVE-22046 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22046.01.patch, HIVE-22046.02.patch, > HIVE-22046.03.patch, HIVE-22046.04.patch, HIVE-22046.05.patch, > HIVE-22046.06.patch, HIVE-22046.07.patch, HIVE-22046.08.patch, > HIVE-22046.09.patch, HIVE-22046.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The goal is to avoid computation of column stats by engines to step on each > other, e.g., Hive and Impala. In longer term, we may introduce a common > representation for the column statistics stored by different engines. > For this issue, we will add a new column 'engine' to TAB_COL_STATS HMS table > (unpartitioned tables) and to PART_COL_STATS HMS table (partitioned tables). > This will prevent conflicts at the column level stats. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22074) Slow compilation due to IN to OR transformation
[
https://issues.apache.org/jira/browse/HIVE-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900688#comment-16900688
]
Hive QA commented on HIVE-22074:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 2 new + 260 unchanged - 1
fixed = 262 total (was 261) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18266/dev-support/hive-personality.sh
|
| git revision | master / 4510efd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18266/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18266/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Slow compilation due to IN to OR transformation
> ---
>
> Key: HIVE-22074
> URL: https://issues.apache.org/jira/browse/HIVE-22074
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch,
> HIVE-22074.3.patch
>
>
> Currently Hive transform IN expressions to OR to apply various CBO rules.
> This incur significant performance hit if IN consist of large number of
> expressions.
> It is better to not transform IN expressions to OR in such cases because
> overall benefit of various optimizations/transformations is unrealized due to
> the compilation overhead
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
