[jira] [Updated] (HIVE-19298) Fix operator tree of CTAS for Druid Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-19298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-19298:
Resolution: Fixed
Target Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Slim!
[~vgarg] Please consider it for 3.0 ?
> Fix operator tree of CTAS for Druid Storage Handler
> ---
>
> Key: HIVE-19298
> URL: https://issues.apache.org/jira/browse/HIVE-19298
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19298.2.patch, HIVE-19298.3.patch,
> HIVE-19298.patch, HIVE-19298.patch
>
>
> Current operator plan of CTAS for Druid storage handler is broken when used
> enables the property \{code} hive.exec.parallel\{code} as \{code} true\{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19357) Vectorization: assert_true HiveException erroneously gets suppressed to NULL
[
https://issues.apache.org/jira/browse/HIVE-19357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466898#comment-16466898
]
Hive QA commented on HIVE-19357:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922312/HIVE-19357.05.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 219 failed/errored test(s), 13539 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Updated] (HIVE-19173) Add Storage Handler runtime information as part of DESCRIBE EXTENDED
[ https://issues.apache.org/jira/browse/HIVE-19173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19173: Resolution: Fixed Fix Version/s: 3.1.0 Target Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Nishant! [~vgarg] Please consider for branch-3 > Add Storage Handler runtime information as part of DESCRIBE EXTENDED > > > Key: HIVE-19173 > URL: https://issues.apache.org/jira/browse/HIVE-19173 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19173.01.patch, HIVE-19173.02.patch > > > Follow up for https://issues.apache.org/jira/browse/HIVE-18976 > Kafka Indexing Service in Druid has a runtime state associated with it. > Druid publishes this runtime state as KafkaSupervisorReport which has latest > offsets as reported by Kafka, the consumer lag per partition, as well as the > aggregate lag of all partitions. > This information is quite useful to know whether a kafka-indexing-service > backed table has latest info or not. > This task is to add a this information as part of the output of DESCRIBE > EXTENDED statement -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19451) Druid Query Execution fails with ClassNotFoundException org.antlr.v4.runtime.CharStream
[
https://issues.apache.org/jira/browse/HIVE-19451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466882#comment-16466882
]
Nishant Bangarwa commented on HIVE-19451:
-
+cc [~ashutoshc] Attached patch fixes the problem by adding the org.antlr
classes, It will unblock the failures,
Also created follow up JIRA to avoid druid query serde -
https://issues.apache.org/jira/browse/HIVE-19452
> Druid Query Execution fails with ClassNotFoundException
> org.antlr.v4.runtime.CharStream
> ---
>
> Key: HIVE-19451
> URL: https://issues.apache.org/jira/browse/HIVE-19451
> Project: Hive
> Issue Type: Task
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-19451.patch
>
>
> Stack trace -
> {code}
> ERROR : Status: Failed
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1524814504173_1344_45_00, diagnostics=[Task failed,
> taskId=task_1524814504173_1344_45_00_29, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1524814504173_1344_45_00_29_0:java.lang.RuntimeException:
> java.io.IOException:
> org.apache.hive.druid.com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
> Cannot construct instance of
> `org.apache.hive.druid.io.druid.segment.virtual.ExpressionVirtualColumn`,
> problem: org/antlr/v4/runtime/CharStream
> at [Source:
> (String)"{"queryType":"scan","dataSource":{"type":"table","name":"tpcds_real_bin_partitioned_orc_1000.tpcds_denormalized_druid_table_7mcd"},"intervals":{"type":"segments","segments":[{"itvl":"1998-11-30T00:00:00.000Z/1998-12-01T00:00:00.000Z","ver":"2018-05-03T11:35:22.230Z","part":0}]},"virtualColumns":[{"type":"expression","name":"vc","expression":"\"__time\"","outputType":"LONG"}],"resultFormat":"compactedList","batchSize":20480,"limit":9223372036854775807,"filter":{"type":"bound","dimension":"i_brand"[truncated
> 241 chars]; line: 1, column: 376] (through reference chain:
> org.apache.hive.druid.io.druid.query.scan.ScanQuery["virtualColumns"]->java.util.ArrayList[0])
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException:
> org.apache.hive.druid.com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
> Cannot construct instance of
> `org.apache.hive.druid.io.druid.segment.virtual.ExpressionVirtualColumn`,
> problem: org/antlr/v4/runtime/CharStream
> at [Source:
> (String)"{"queryType":"scan","dataSource":{"type":"table","name":"tpcds_real_bin_partitioned_orc_1000.tpcds_denormalized_druid_table_7mcd"},"intervals":{"type":"segments","segments":[{"itvl":"1998-11-30T00:00:00.000Z/1998-12-01T00:00:00.000Z","ver":"2018-05-03T11:35:22.230Z","part":0}]},"virtualColumns":[{"type":"expression","name":"vc","expression":"\"__time\"","outputType":"LONG"}],"resultFormat":"compactedList","batchSize":20480,"limit":9223372036854775807,"filter":{"type":"bound","dimension":"i_brand"[truncated
> 241 chars]; line: 1, column: 376] (through reference chain:
> org.apache.hive.druid.io.druid.query.scan.ScanQuery["virtualColumns"]->java.util.ArrayList[0])
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at
>
[jira] [Assigned] (HIVE-19452) Avoid Deserializing and Serializing Druid query in DruidRecordReaders
[ https://issues.apache.org/jira/browse/HIVE-19452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa reassigned HIVE-19452: --- > Avoid Deserializing and Serializing Druid query in DruidRecordReaders > - > > Key: HIVE-19452 > URL: https://issues.apache.org/jira/browse/HIVE-19452 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > > Druid record reader deserializes and serializes the Druid query before > sending it to druid. > This can be avoided and we can stop packaging some of druid dependencies e.g. > org.antlr from druid-handler selfcontained jar. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19451) Druid Query Execution fails with ClassNotFoundException org.antlr.v4.runtime.CharStream
[
https://issues.apache.org/jira/browse/HIVE-19451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-19451:
Status: Patch Available (was: Open)
> Druid Query Execution fails with ClassNotFoundException
> org.antlr.v4.runtime.CharStream
> ---
>
> Key: HIVE-19451
> URL: https://issues.apache.org/jira/browse/HIVE-19451
> Project: Hive
> Issue Type: Task
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-19451.patch
>
>
> Stack trace -
> {code}
> ERROR : Status: Failed
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1524814504173_1344_45_00, diagnostics=[Task failed,
> taskId=task_1524814504173_1344_45_00_29, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1524814504173_1344_45_00_29_0:java.lang.RuntimeException:
> java.io.IOException:
> org.apache.hive.druid.com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
> Cannot construct instance of
> `org.apache.hive.druid.io.druid.segment.virtual.ExpressionVirtualColumn`,
> problem: org/antlr/v4/runtime/CharStream
> at [Source:
> (String)"{"queryType":"scan","dataSource":{"type":"table","name":"tpcds_real_bin_partitioned_orc_1000.tpcds_denormalized_druid_table_7mcd"},"intervals":{"type":"segments","segments":[{"itvl":"1998-11-30T00:00:00.000Z/1998-12-01T00:00:00.000Z","ver":"2018-05-03T11:35:22.230Z","part":0}]},"virtualColumns":[{"type":"expression","name":"vc","expression":"\"__time\"","outputType":"LONG"}],"resultFormat":"compactedList","batchSize":20480,"limit":9223372036854775807,"filter":{"type":"bound","dimension":"i_brand"[truncated
> 241 chars]; line: 1, column: 376] (through reference chain:
> org.apache.hive.druid.io.druid.query.scan.ScanQuery["virtualColumns"]->java.util.ArrayList[0])
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException:
> org.apache.hive.druid.com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
> Cannot construct instance of
> `org.apache.hive.druid.io.druid.segment.virtual.ExpressionVirtualColumn`,
> problem: org/antlr/v4/runtime/CharStream
> at [Source:
> (String)"{"queryType":"scan","dataSource":{"type":"table","name":"tpcds_real_bin_partitioned_orc_1000.tpcds_denormalized_druid_table_7mcd"},"intervals":{"type":"segments","segments":[{"itvl":"1998-11-30T00:00:00.000Z/1998-12-01T00:00:00.000Z","ver":"2018-05-03T11:35:22.230Z","part":0}]},"virtualColumns":[{"type":"expression","name":"vc","expression":"\"__time\"","outputType":"LONG"}],"resultFormat":"compactedList","batchSize":20480,"limit":9223372036854775807,"filter":{"type":"bound","dimension":"i_brand"[truncated
> 241 chars]; line: 1, column: 376] (through reference chain:
> org.apache.hive.druid.io.druid.query.scan.ScanQuery["virtualColumns"]->java.util.ArrayList[0])
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:438)
> at
>
[jira] [Updated] (HIVE-19451) Druid Query Execution fails with ClassNotFoundException org.antlr.v4.runtime.CharStream
[
https://issues.apache.org/jira/browse/HIVE-19451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-19451:
Attachment: HIVE-19451.patch
> Druid Query Execution fails with ClassNotFoundException
> org.antlr.v4.runtime.CharStream
> ---
>
> Key: HIVE-19451
> URL: https://issues.apache.org/jira/browse/HIVE-19451
> Project: Hive
> Issue Type: Task
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-19451.patch
>
>
> Stack trace -
> {code}
> ERROR : Status: Failed
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1524814504173_1344_45_00, diagnostics=[Task failed,
> taskId=task_1524814504173_1344_45_00_29, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1524814504173_1344_45_00_29_0:java.lang.RuntimeException:
> java.io.IOException:
> org.apache.hive.druid.com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
> Cannot construct instance of
> `org.apache.hive.druid.io.druid.segment.virtual.ExpressionVirtualColumn`,
> problem: org/antlr/v4/runtime/CharStream
> at [Source:
> (String)"{"queryType":"scan","dataSource":{"type":"table","name":"tpcds_real_bin_partitioned_orc_1000.tpcds_denormalized_druid_table_7mcd"},"intervals":{"type":"segments","segments":[{"itvl":"1998-11-30T00:00:00.000Z/1998-12-01T00:00:00.000Z","ver":"2018-05-03T11:35:22.230Z","part":0}]},"virtualColumns":[{"type":"expression","name":"vc","expression":"\"__time\"","outputType":"LONG"}],"resultFormat":"compactedList","batchSize":20480,"limit":9223372036854775807,"filter":{"type":"bound","dimension":"i_brand"[truncated
> 241 chars]; line: 1, column: 376] (through reference chain:
> org.apache.hive.druid.io.druid.query.scan.ScanQuery["virtualColumns"]->java.util.ArrayList[0])
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException:
> org.apache.hive.druid.com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
> Cannot construct instance of
> `org.apache.hive.druid.io.druid.segment.virtual.ExpressionVirtualColumn`,
> problem: org/antlr/v4/runtime/CharStream
> at [Source:
> (String)"{"queryType":"scan","dataSource":{"type":"table","name":"tpcds_real_bin_partitioned_orc_1000.tpcds_denormalized_druid_table_7mcd"},"intervals":{"type":"segments","segments":[{"itvl":"1998-11-30T00:00:00.000Z/1998-12-01T00:00:00.000Z","ver":"2018-05-03T11:35:22.230Z","part":0}]},"virtualColumns":[{"type":"expression","name":"vc","expression":"\"__time\"","outputType":"LONG"}],"resultFormat":"compactedList","batchSize":20480,"limit":9223372036854775807,"filter":{"type":"bound","dimension":"i_brand"[truncated
> 241 chars]; line: 1, column: 376] (through reference chain:
> org.apache.hive.druid.io.druid.query.scan.ScanQuery["virtualColumns"]->java.util.ArrayList[0])
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:438)
> at
>
[jira] [Assigned] (HIVE-19451) Druid Query Execution fails with ClassNotFoundException org.antlr.v4.runtime.CharStream
[
https://issues.apache.org/jira/browse/HIVE-19451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa reassigned HIVE-19451:
---
> Druid Query Execution fails with ClassNotFoundException
> org.antlr.v4.runtime.CharStream
> ---
>
> Key: HIVE-19451
> URL: https://issues.apache.org/jira/browse/HIVE-19451
> Project: Hive
> Issue Type: Task
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>
> Stack trace -
> {code}
> ERROR : Status: Failed
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1524814504173_1344_45_00, diagnostics=[Task failed,
> taskId=task_1524814504173_1344_45_00_29, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1524814504173_1344_45_00_29_0:java.lang.RuntimeException:
> java.io.IOException:
> org.apache.hive.druid.com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
> Cannot construct instance of
> `org.apache.hive.druid.io.druid.segment.virtual.ExpressionVirtualColumn`,
> problem: org/antlr/v4/runtime/CharStream
> at [Source:
> (String)"{"queryType":"scan","dataSource":{"type":"table","name":"tpcds_real_bin_partitioned_orc_1000.tpcds_denormalized_druid_table_7mcd"},"intervals":{"type":"segments","segments":[{"itvl":"1998-11-30T00:00:00.000Z/1998-12-01T00:00:00.000Z","ver":"2018-05-03T11:35:22.230Z","part":0}]},"virtualColumns":[{"type":"expression","name":"vc","expression":"\"__time\"","outputType":"LONG"}],"resultFormat":"compactedList","batchSize":20480,"limit":9223372036854775807,"filter":{"type":"bound","dimension":"i_brand"[truncated
> 241 chars]; line: 1, column: 376] (through reference chain:
> org.apache.hive.druid.io.druid.query.scan.ScanQuery["virtualColumns"]->java.util.ArrayList[0])
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException:
> org.apache.hive.druid.com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
> Cannot construct instance of
> `org.apache.hive.druid.io.druid.segment.virtual.ExpressionVirtualColumn`,
> problem: org/antlr/v4/runtime/CharStream
> at [Source:
> (String)"{"queryType":"scan","dataSource":{"type":"table","name":"tpcds_real_bin_partitioned_orc_1000.tpcds_denormalized_druid_table_7mcd"},"intervals":{"type":"segments","segments":[{"itvl":"1998-11-30T00:00:00.000Z/1998-12-01T00:00:00.000Z","ver":"2018-05-03T11:35:22.230Z","part":0}]},"virtualColumns":[{"type":"expression","name":"vc","expression":"\"__time\"","outputType":"LONG"}],"resultFormat":"compactedList","batchSize":20480,"limit":9223372036854775807,"filter":{"type":"bound","dimension":"i_brand"[truncated
> 241 chars]; line: 1, column: 376] (through reference chain:
> org.apache.hive.druid.io.druid.query.scan.ScanQuery["virtualColumns"]->java.util.ArrayList[0])
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:438)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:157)
> at
>
[jira] [Commented] (HIVE-19370) Issue: ADD Months function on timestamp datatype fields in hive
[ https://issues.apache.org/jira/browse/HIVE-19370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466874#comment-16466874 ] Amit Chauhan commented on HIVE-19370: - [~bharos92] But imagine a scenario where we have data coming in and the difference between them is in hours minutes or seconds. if I use add months on same all date values will be having then same year month date values and with no difference. any thought on same ? > Issue: ADD Months function on timestamp datatype fields in hive > --- > > Key: HIVE-19370 > URL: https://issues.apache.org/jira/browse/HIVE-19370 > Project: Hive > Issue Type: Bug >Reporter: Amit Chauhan >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > *Issue:* > while using ADD_Months function on a timestamp datatype column the output > omits the time part[HH:MM:SS] part from output. > which should not be the case. > *query:* EMAIL_FAILURE_DTMZ is of datatype timestamp in hive. > hive> select CUSTOMER_ID,EMAIL_FAILURE_DTMZ,ADD_MONTHS (EMAIL_FAILURE_DTMZ , > 1) from TABLE1 where CUSTOMER_ID=125674937; > OK > 125674937 2015-12-09 12:25:53 2016-01-09 > *hiver version :* > hive> !hive --version; > Hive 1.2.1000.2.5.6.0-40 > > can you please help if somehow I can get below as output: > > 125674937 2015-12-09 12:25:53 2016-01-09 12:25:53 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19370) Issue: ADD Months function on timestamp datatype fields in hive
[
https://issues.apache.org/jira/browse/HIVE-19370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466870#comment-16466870
]
Bharathkrishna Guruvayoor Murali commented on HIVE-19370:
-
I do not think this is a bug, as it is mentioned in the add_months description
in [LanguageManual
UDF|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF] :
{quote}start_date is a string, date or timestamp. num_months is an integer. The
time part of start_date is ignored.
{quote}
> Issue: ADD Months function on timestamp datatype fields in hive
> ---
>
> Key: HIVE-19370
> URL: https://issues.apache.org/jira/browse/HIVE-19370
> Project: Hive
> Issue Type: Bug
>Reporter: Amit Chauhan
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> *Issue:*
> while using ADD_Months function on a timestamp datatype column the output
> omits the time part[HH:MM:SS] part from output.
> which should not be the case.
> *query:* EMAIL_FAILURE_DTMZ is of datatype timestamp in hive.
> hive> select CUSTOMER_ID,EMAIL_FAILURE_DTMZ,ADD_MONTHS (EMAIL_FAILURE_DTMZ ,
> 1) from TABLE1 where CUSTOMER_ID=125674937;
> OK
> 125674937 2015-12-09 12:25:53 2016-01-09
> *hiver version :*
> hive> !hive --version;
> Hive 1.2.1000.2.5.6.0-40
>
> can you please help if somehow I can get below as output:
>
> 125674937 2015-12-09 12:25:53 2016-01-09 12:25:53
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19357) Vectorization: assert_true HiveException erroneously gets suppressed to NULL
[
https://issues.apache.org/jira/browse/HIVE-19357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466866#comment-16466866
]
Hive QA commented on HIVE-19357:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
55s{color} | {color:red} branch/itests/hive-jmh cannot run convertXmlToText
from findbugs {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2323 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} common: The patch generated 1 new + 426 unchanged - 0
fixed = 427 total (was 426) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
28s{color} | {color:red} ql: The patch generated 2 new + 4605 unchanged - 9
fixed = 4607 total (was 4614) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 3 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 9m
46s{color} | {color:red} patch/itests/hive-jmh cannot run convertXmlToText from
findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10756/dev-support/hive-personality.sh
|
| git revision | master / 0930aec |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10756/yetus/branch-findbugs-itests_hive-jmh.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10756/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10756/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10756/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10756/yetus/whitespace-tabs.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10756/yetus/patch-findbugs-itests_hive-jmh.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10756/yetus/patch-asflicense-problems.txt
|
| modules | C: common itests/hive-jmh ql U: . |
| Console output |
[jira] [Commented] (HIVE-19248) REPL LOAD couldn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466860#comment-16466860
]
Sankar Hariappan commented on HIVE-19248:
-
Thanks for the review [~maheshk114]!
[~thejas], could you please review and +1 the patch?
> REPL LOAD couldn't copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Blocker
> Labels: DR, pull-request-available, replication
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19248.01.patch, HIVE-19248.02.patch
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> Distcp failed as the CM path for the file doesn't point to source file
> system. So, it is needed to get the qualified cm root URI as part of files
> listed in dump.
> Also, REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, then retry with same set
> of files again but need to set CM path if original source file is missing or
> modified based on checksum. Let distcp to skip the properly copied files.
> FileUtil.copy will always overwrite the files.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This message was
[jira] [Commented] (HIVE-19334) Use actual file size rather than stats for fetch task optimization with external tables
[
https://issues.apache.org/jira/browse/HIVE-19334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466832#comment-16466832
]
Hive QA commented on HIVE-19334:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922346/HIVE-19334.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 216 failed/errored test(s), 13538 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-19248) REPL LOAD couldn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466829#comment-16466829
]
mahesh kumar behera commented on HIVE-19248:
HIVE-19248.02.patch looks fine to me
> REPL LOAD couldn't copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Blocker
> Labels: DR, pull-request-available, replication
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19248.01.patch, HIVE-19248.02.patch
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> Distcp failed as the CM path for the file doesn't point to source file
> system. So, it is needed to get the qualified cm root URI as part of files
> listed in dump.
> Also, REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, then retry with same set
> of files again but need to set CM path if original source file is missing or
> modified based on checksum. Let distcp to skip the properly copied files.
> FileUtil.copy will always overwrite the files.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19435) Incremental replication cause data loss if a table is dropped followed by create and insert-into with different partition type.
[ https://issues.apache.org/jira/browse/HIVE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19435: Status: Patch Available (was: Open) Attached 03.patch for conflicts with master. > Incremental replication cause data loss if a table is dropped followed by > create and insert-into with different partition type. > --- > > Key: HIVE-19435 > URL: https://issues.apache.org/jira/browse/HIVE-19435 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19435.01.patch, HIVE-19435.02.patch, > HIVE-19435.03.patch > > > If the incremental dump have drop of partitioned table followed by > create/insert on non-partitioned table with same name, doesn't replicate the > data. Explained below. > Let's say we have a partitioned table T1 which was already replicated to > target. > DROP_TABLE(T1)->CREATE_TABLE(T1) (Non-partitioned) -> INSERT(T1)(10) > After REPL LOAD, T1 doesn't have any data. > Same is valid for non-partitioned to partitioned and partition spec mismatch > case as well. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19435) Incremental replication cause data loss if a table is dropped followed by create and insert-into with different partition type.
[ https://issues.apache.org/jira/browse/HIVE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19435: Status: Open (was: Patch Available) > Incremental replication cause data loss if a table is dropped followed by > create and insert-into with different partition type. > --- > > Key: HIVE-19435 > URL: https://issues.apache.org/jira/browse/HIVE-19435 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19435.01.patch, HIVE-19435.02.patch, > HIVE-19435.03.patch > > > If the incremental dump have drop of partitioned table followed by > create/insert on non-partitioned table with same name, doesn't replicate the > data. Explained below. > Let's say we have a partitioned table T1 which was already replicated to > target. > DROP_TABLE(T1)->CREATE_TABLE(T1) (Non-partitioned) -> INSERT(T1)(10) > After REPL LOAD, T1 doesn't have any data. > Same is valid for non-partitioned to partitioned and partition spec mismatch > case as well. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19435) Incremental replication cause data loss if a table is dropped followed by create and insert-into with different partition type.
[ https://issues.apache.org/jira/browse/HIVE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19435: Attachment: HIVE-19435.03.patch > Incremental replication cause data loss if a table is dropped followed by > create and insert-into with different partition type. > --- > > Key: HIVE-19435 > URL: https://issues.apache.org/jira/browse/HIVE-19435 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19435.01.patch, HIVE-19435.02.patch, > HIVE-19435.03.patch > > > If the incremental dump have drop of partitioned table followed by > create/insert on non-partitioned table with same name, doesn't replicate the > data. Explained below. > Let's say we have a partitioned table T1 which was already replicated to > target. > DROP_TABLE(T1)->CREATE_TABLE(T1) (Non-partitioned) -> INSERT(T1)(10) > After REPL LOAD, T1 doesn't have any data. > Same is valid for non-partitioned to partitioned and partition spec mismatch > case as well. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19287) parse error with semicolon in comment which starts with whitespace in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19287: -- Status: Patch Available (was: Open) resubmit the patch to trigger the pre-merge check > parse error with semicolon in comment which starts with whitespace in file > -- > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch, HIVE-19287.2.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-18871: -- Status: Open (was: Patch Available) > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-18871.1.patch, HIVE-18871.2.patch, > HIVE-18871.3.patch, HIVE-18871.4.patch > > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) >
[jira] [Updated] (HIVE-19287) parse error with semicolon in comment which starts with whitespace in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19287: -- Status: Open (was: Patch Available) > parse error with semicolon in comment which starts with whitespace in file > -- > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch, HIVE-19287.2.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-18871: -- Status: Patch Available (was: Open) > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-18871.1.patch, HIVE-18871.2.patch, > HIVE-18871.3.patch, HIVE-18871.4.patch > > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) >
[jira] [Commented] (HIVE-19202) CBO failed due to NullPointerException in HiveAggregate.isBucketedInput()
[ https://issues.apache.org/jira/browse/HIVE-19202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466817#comment-16466817 ] zhuwei commented on HIVE-19202: --- [~dvoros] The query which failing is in our production environment and a little bit complicate. According to the code logic, there are some keyword related: count/sum/group by/join > CBO failed due to NullPointerException in HiveAggregate.isBucketedInput() > - > > Key: HIVE-19202 > URL: https://issues.apache.org/jira/browse/HIVE-19202 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Critical > Fix For: 3.1.0 > > Attachments: HIVE-19202.1.patch, HIVE-19202.2.patch > > > I ran a query with join and group by with below settings, COB failed due to > NullPointerException in HiveAggregate.isBucketedInput() > set hive.execution.engine=tez; > set hive.cbo.costmodel.extended=true; > > In class HiveRelMdDistribution, we implemented below functions: > public RelDistribution distribution(HiveAggregate aggregate, RelMetadataQuery > mq) > public RelDistribution distribution(HiveJoin join, RelMetadataQuery mq) > > But in HiveAggregate.isBucketedInput, the argument passed to distribution is > "this.getInput()" > , obviously it's not right here. The right argument needed is "this" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19248) REPL LOAD couldn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-19248:
Priority: Blocker (was: Major)
> REPL LOAD couldn't copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Blocker
> Labels: DR, pull-request-available, replication
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19248.01.patch, HIVE-19248.02.patch
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> Distcp failed as the CM path for the file doesn't point to source file
> system. So, it is needed to get the qualified cm root URI as part of files
> listed in dump.
> Also, REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, then retry with same set
> of files again but need to set CM path if original source file is missing or
> modified based on checksum. Let distcp to skip the properly copied files.
> FileUtil.copy will always overwrite the files.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466816#comment-16466816 ] Rui Li commented on HIVE-18533: --- Hi [~stakiar], my concern for SparkLauncherFuture is mainly about the cancel logic. SparkLauncherFuture::cancel calls SparkAppHandle::stop, which I think is an async method. So it doesn't immediately unblock threads waiting on SparkLauncherFuture::get. And subsequent calls to isCancelled and isDone may not return true. Besides, JavaDoc mentions SparkAppHandle::stop is only best effort to ask the app to stop, so it doesn't even guarantee a state change. Another issue is SparkLauncherFuture::isCancelled considers all failed states as cancelled. So it may return true even if cancel is not called. I know this might not be an issue according to the way AbstractSparkClient works at the moment. But if we want to make changes to AbstractSparkClient in the future, it's better if the two sub-classes behave consistently and both honor the Future contracts. If we use a FutureTask, we can interrupt the thread when we cancel the Future. The thread can handle the interrupt exception and call SparkAppHandle::stop (probably need to cancel the RPC as well) -- similar to what we do in SparkSubmitSparkClient. > Add option to use InProcessLauncher to submit spark jobs > > > Key: HIVE-18533 > URL: https://issues.apache.org/jira/browse/HIVE-18533 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18533.1.patch, HIVE-18533.2.patch, > HIVE-18533.3.patch, HIVE-18533.4.patch, HIVE-18533.5.patch, > HIVE-18533.6.patch, HIVE-18533.7.patch, HIVE-18533.8.patch, > HIVE-18533.9.patch, HIVE-18533.91.patch, HIVE-18831.93.patch > > > See discussion in HIVE-16484 for details. > I think this will help with reducing the amount of time it takes to open a > HoS session + debuggability (no need launch a separate process to run a Spark > app). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18193: --- Comment: was deleted (was: with Acid replication not yet supported, if we upgrade all tables to ACID, will it not disable replication for all tables ?) > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch, HIVE-18193.02.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466815#comment-16466815 ] mahesh kumar behera commented on HIVE-18193: with Acid replication not yet supported, if we upgrade all tables to ACID, will it not disable replication for all tables ? > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch, HIVE-18193.02.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19450) OOM due to map join and backup task not invoked
[ https://issues.apache.org/jira/browse/HIVE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19450: -- Status: Patch Available (was: Open) The fix is to further checkout parent task's backup task. > OOM due to map join and backup task not invoked > --- > > Key: HIVE-19450 > URL: https://issues.apache.org/jira/browse/HIVE-19450 > Project: Hive > Issue Type: Bug >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-19450.1.patch > > > Map join task may cause OOM due to orc compression , in most cases , a backup > task will be invoked. However , if the size of hash table is close to memory > limit, the task which load the hash table will NOT fail . OOM will happen in > next task witch do local join. The load task has a backup but next task not. > So in this case , the whole query will fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19450) OOM due to map join and backup task not invoked
[ https://issues.apache.org/jira/browse/HIVE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19450: -- Attachment: HIVE-19450.1.patch > OOM due to map join and backup task not invoked > --- > > Key: HIVE-19450 > URL: https://issues.apache.org/jira/browse/HIVE-19450 > Project: Hive > Issue Type: Bug >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-19450.1.patch > > > Map join task may cause OOM due to orc compression , in most cases , a backup > task will be invoked. However , if the size of hash table is close to memory > limit, the task which load the hash table will NOT fail . OOM will happen in > next task witch do local join. The load task has a backup but next task not. > So in this case , the whole query will fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19450) OOM due to map join and backup task not invoked
[ https://issues.apache.org/jira/browse/HIVE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei reassigned HIVE-19450: - > OOM due to map join and backup task not invoked > --- > > Key: HIVE-19450 > URL: https://issues.apache.org/jira/browse/HIVE-19450 > Project: Hive > Issue Type: Bug >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > > Map join task may cause OOM due to orc compression , in most cases , a backup > task will be invoked. However , if the size of hash table is close to memory > limit, the task which load the hash table will NOT fail . OOM will happen in > next task witch do local join. The load task has a backup but next task not. > So in this case , the whole query will fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19334) Use actual file size rather than stats for fetch task optimization with external tables
[
https://issues.apache.org/jira/browse/HIVE-19334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466782#comment-16466782
]
Hive QA commented on HIVE-19334:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 2323 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10755/dev-support/hive-personality.sh
|
| git revision | master / 0930aec |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10755/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10755/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use actual file size rather than stats for fetch task optimization with
> external tables
> ---
>
> Key: HIVE-19334
> URL: https://issues.apache.org/jira/browse/HIVE-19334
> Project: Hive
> Issue Type: Sub-task
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-19334.1.patch, HIVE-19334.2.patch,
> HIVE-19334.3.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19448) Vectorization: sysdb test doesn't work after enabling vectorization by default
[
https://issues.apache.org/jira/browse/HIVE-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19448:
Attachment: HIVE-19448.01.patch
> Vectorization: sysdb test doesn't work after enabling vectorization by default
> --
>
> Key: HIVE-19448
> URL: https://issues.apache.org/jira/browse/HIVE-19448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19448.01.patch
>
>
> Solution: EXCLUDE org.apache.hive.storage.jdbc.JdbcInputFormat which returns
> String for boolean.
> {noformat}
> Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
> java.lang.Boolean
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaBooleanObjectInspector.getPrimitiveWritableObject(JavaBooleanObjectInspector.java:36)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:434)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:347)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:948){noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19448) Vectorization: sysdb test doesn't work after enabling vectorization by default
[
https://issues.apache.org/jira/browse/HIVE-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19448:
Status: Patch Available (was: Open)
> Vectorization: sysdb test doesn't work after enabling vectorization by default
> --
>
> Key: HIVE-19448
> URL: https://issues.apache.org/jira/browse/HIVE-19448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19448.01.patch
>
>
> Solution: EXCLUDE org.apache.hive.storage.jdbc.JdbcInputFormat which returns
> String for boolean.
> {noformat}
> Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
> java.lang.Boolean
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaBooleanObjectInspector.getPrimitiveWritableObject(JavaBooleanObjectInspector.java:36)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:434)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:347)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:948){noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19448) Vectorization: sysdb test doesn't work after enabling vectorization by default
[
https://issues.apache.org/jira/browse/HIVE-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19448:
Description:
Solution: EXCLUDE org.apache.hive.storage.jdbc.JdbcInputFormat which returns
String for boolean.
{noformat}
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
java.lang.Boolean
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaBooleanObjectInspector.getPrimitiveWritableObject(JavaBooleanObjectInspector.java:36)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:434)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:347)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:948){noformat}
was:
{noformat}
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
java.lang.Boolean
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaBooleanObjectInspector.getPrimitiveWritableObject(JavaBooleanObjectInspector.java:36)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:434)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:347)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:948){noformat}
> Vectorization: sysdb test doesn't work after enabling vectorization by default
> --
>
> Key: HIVE-19448
> URL: https://issues.apache.org/jira/browse/HIVE-19448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>
> Solution: EXCLUDE org.apache.hive.storage.jdbc.JdbcInputFormat which returns
> String for boolean.
> {noformat}
> Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
> java.lang.Boolean
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaBooleanObjectInspector.getPrimitiveWritableObject(JavaBooleanObjectInspector.java:36)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:434)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:347)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:948){noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19439) MapWork shouldn't be reused when Spark task fails during initialization
[ https://issues.apache.org/jira/browse/HIVE-19439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466775#comment-16466775 ] Rui Li commented on HIVE-19439: --- BTW, the hash table is loaded when we init the dummy operators [here|https://github.com/apache/hive/blob/rel/release-2.2.0/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java#L113]. > MapWork shouldn't be reused when Spark task fails during initialization > --- > > Key: HIVE-19439 > URL: https://issues.apache.org/jira/browse/HIVE-19439 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Rui Li >Priority: Major > > Issue identified in HIVE-19388. When a Spark task fails during initializing > the map operator, the task is retried with the same MapWork retrieved from > cache. This can be problematic because the MapWork may be partially > initialized, e.g. some operators are already in INIT state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19439) MapWork shouldn't be reused when Spark task fails during initialization
[ https://issues.apache.org/jira/browse/HIVE-19439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466772#comment-16466772 ] Rui Li commented on HIVE-19439: --- Hi [~vihangk1], the task is retried by Spark, and it calls SparkMapRecordHandler::init to initialize the map operator. This is where we retrieve the MapWork [from cache|https://github.com/apache/hive/blob/rel/release-2.2.0/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java#L75]. I'm not sure whether we have a way to reset the operators to UNINIT state. If not, guess we have to clear the cache when initialization fails. > MapWork shouldn't be reused when Spark task fails during initialization > --- > > Key: HIVE-19439 > URL: https://issues.apache.org/jira/browse/HIVE-19439 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Rui Li >Priority: Major > > Issue identified in HIVE-19388. When a Spark task fails during initializing > the map operator, the task is retried with the same MapWork retrieved from > cache. This can be problematic because the MapWork may be partially > initialized, e.g. some operators are already in INIT state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19449) Create minimized uber jar for hive streaming module
[ https://issues.apache.org/jira/browse/HIVE-19449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-19449: Assignee: Prasanth Jayachandran > Create minimized uber jar for hive streaming module > --- > > Key: HIVE-19449 > URL: https://issues.apache.org/jira/browse/HIVE-19449 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > Hive streaming API depends on several hive modules (common, serde, ql, orc, > standalone-metastore etc). Users of the API has to include all the > dependencies in the classpath for it to work correctly. Provide a uber jar > with minimal set of dependencies that are required to make use of new > streaming API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18052) Run p-tests on mm tables
[
https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466735#comment-16466735
]
Hive QA commented on HIVE-18052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} hcatalog/core in master has 33 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/webhcat/java-client in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 2323 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} service in master has 50 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
59s{color} | {color:blue} standalone-metastore in master has 215 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
58s{color} | {color:red} root: The patch generated 13 new + 2164 unchanged - 9
fixed = 2177 total (was 2173) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch core passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch hcatalog-pig-adapter passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch streaming passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hcatalog/webhcat/java-client: The patch generated 0
new + 108 unchanged - 1 fixed = 108 total (was 109) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} itests/hive-unit: The patch generated 7 new + 852
unchanged - 7 fixed = 859 total (was 859) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 3 new + 156 unchanged - 0
fixed = 159 total (was 156) {color} |
|
[jira] [Updated] (HIVE-19259) Create view on tables having union all fail with "Table not found"
[
https://issues.apache.org/jira/browse/HIVE-19259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19259:
---
Attachment: HIVE-19259.patch
> Create view on tables having union all fail with "Table not found"
> --
>
> Key: HIVE-19259
> URL: https://issues.apache.org/jira/browse/HIVE-19259
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 1.2.1
> Environment: hive-1.2.1
>
>Reporter: Rajkumar Singh
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19259.patch
>
>
> create view on table with union work well while "union all" failed with table
> not found, here are the reproduce steps.
> {code}
> _hive> create table foo(id int);_
> _OK_
> _Time taken: 0.401 seconds_
> _hive> create table bar(id int);_
> _OK_
>
> _// view on table union_
> _hive> create view unionview as with tmp_1 as ( select * from foo ), tmp_2 as
> (select * from bar ) select * from tmp_1 union select * from tmp_2;_
> _OK_
> _Time taken: 0.517 seconds_
> _hive> select * from unionview;_
> _OK_
> _Time taken: 5.805 seconds_
>
>
> _// view on union all_
> _hive> create view unionallview as with tmp_1 as ( select * from foo ), tmp_2
> as (select * from bar ) select * from tmp_1 union all select * from tmp_2;_
> _OK_
> _Time taken: 1.535 seconds_
> _hive> select * from unionallview;_
> _FAILED: SemanticException Line 1:134 Table not found 'tmp_1' in definition
> of VIEW unionallview [_
> _with tmp_1 as ( select `foo`.`id` from `default`.`foo` ), tmp_2 as (select
> `bar`.`id` from `default`.`bar` ) select `tmp_1`.`id` from tmp_1 union all
> select `tmp_2`.`id` from tmp_2_
> _] used as unionallview at Line 1:14_
> _{code}_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-19259) Create view on tables having union all fail with "Table not found"
[
https://issues.apache.org/jira/browse/HIVE-19259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19259 started by Jesus Camacho Rodriguez.
--
> Create view on tables having union all fail with "Table not found"
> --
>
> Key: HIVE-19259
> URL: https://issues.apache.org/jira/browse/HIVE-19259
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 1.2.1
> Environment: hive-1.2.1
>
>Reporter: Rajkumar Singh
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> create view on table with union work well while "union all" failed with table
> not found, here are the reproduce steps.
> {code}
> _hive> create table foo(id int);_
> _OK_
> _Time taken: 0.401 seconds_
> _hive> create table bar(id int);_
> _OK_
>
> _// view on table union_
> _hive> create view unionview as with tmp_1 as ( select * from foo ), tmp_2 as
> (select * from bar ) select * from tmp_1 union select * from tmp_2;_
> _OK_
> _Time taken: 0.517 seconds_
> _hive> select * from unionview;_
> _OK_
> _Time taken: 5.805 seconds_
>
>
> _// view on union all_
> _hive> create view unionallview as with tmp_1 as ( select * from foo ), tmp_2
> as (select * from bar ) select * from tmp_1 union all select * from tmp_2;_
> _OK_
> _Time taken: 1.535 seconds_
> _hive> select * from unionallview;_
> _FAILED: SemanticException Line 1:134 Table not found 'tmp_1' in definition
> of VIEW unionallview [_
> _with tmp_1 as ( select `foo`.`id` from `default`.`foo` ), tmp_2 as (select
> `bar`.`id` from `default`.`bar` ) select `tmp_1`.`id` from tmp_1 union all
> select `tmp_2`.`id` from tmp_2_
> _] used as unionallview at Line 1:14_
> _{code}_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19259) Create view on tables having union all fail with "Table not found"
[
https://issues.apache.org/jira/browse/HIVE-19259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19259:
---
Status: Patch Available (was: In Progress)
> Create view on tables having union all fail with "Table not found"
> --
>
> Key: HIVE-19259
> URL: https://issues.apache.org/jira/browse/HIVE-19259
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 1.2.1
> Environment: hive-1.2.1
>
>Reporter: Rajkumar Singh
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> create view on table with union work well while "union all" failed with table
> not found, here are the reproduce steps.
> {code}
> _hive> create table foo(id int);_
> _OK_
> _Time taken: 0.401 seconds_
> _hive> create table bar(id int);_
> _OK_
>
> _// view on table union_
> _hive> create view unionview as with tmp_1 as ( select * from foo ), tmp_2 as
> (select * from bar ) select * from tmp_1 union select * from tmp_2;_
> _OK_
> _Time taken: 0.517 seconds_
> _hive> select * from unionview;_
> _OK_
> _Time taken: 5.805 seconds_
>
>
> _// view on union all_
> _hive> create view unionallview as with tmp_1 as ( select * from foo ), tmp_2
> as (select * from bar ) select * from tmp_1 union all select * from tmp_2;_
> _OK_
> _Time taken: 1.535 seconds_
> _hive> select * from unionallview;_
> _FAILED: SemanticException Line 1:134 Table not found 'tmp_1' in definition
> of VIEW unionallview [_
> _with tmp_1 as ( select `foo`.`id` from `default`.`foo` ), tmp_2 as (select
> `bar`.`id` from `default`.`bar` ) select `tmp_1`.`id` from tmp_1 union all
> select `tmp_2`.`id` from tmp_2_
> _] used as unionallview at Line 1:14_
> _{code}_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)
[ https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466697#comment-16466697 ] Sergey Shelukhin edited comment on HIVE-19258 at 5/8/18 1:12 AM: - Rebased the patch, addressed CR feedback (including making the test work for now without the text format case covered; that requires the MR fix), integrated with exim changes (and also BHIF changes), expanded the test to cover those. [~ashutoshc] can you take a look at the updates? was (Author: sershe): Rebased the patch, addressed CR feedback (including making the test work for now without the text format case covered; that requires the MR fix), integrated with exim changes (and also BHIF changes), expanded the test to cover those. > add originals support to MM tables (and make the conversion a metadata only > operation) > -- > > Key: HIVE-19258 > URL: https://issues.apache.org/jira/browse/HIVE-19258 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19258.01.patch, HIVE-19258.02.patch, > HIVE-19258.03.patch, HIVE-19258.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)
[ https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466697#comment-16466697 ] Sergey Shelukhin edited comment on HIVE-19258 at 5/8/18 1:10 AM: - Rebased the patch, addressed CR feedback (including making the test work for now without the text format case covered; that requires the MR fix), integrated with exim changes (and also BHIF changes), expanded the test to cover those. was (Author: sershe): Rebased the patch, addressed CR feedback (including making the test work for now without the text format case covered; that requires the MR fix), integrated with exim changes, expanded the test to cover those. > add originals support to MM tables (and make the conversion a metadata only > operation) > -- > > Key: HIVE-19258 > URL: https://issues.apache.org/jira/browse/HIVE-19258 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19258.01.patch, HIVE-19258.02.patch, > HIVE-19258.03.patch, HIVE-19258.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)
[ https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466697#comment-16466697 ] Sergey Shelukhin commented on HIVE-19258: - Rebased the patch, addressed CR feedback (including making the test work for now without the text format case covered; that requires the MR fix), integrated with exim changes, expanded the test to cover those. > add originals support to MM tables (and make the conversion a metadata only > operation) > -- > > Key: HIVE-19258 > URL: https://issues.apache.org/jira/browse/HIVE-19258 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19258.01.patch, HIVE-19258.02.patch, > HIVE-19258.03.patch, HIVE-19258.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)
[ https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19258: Attachment: HIVE-19258.03.patch > add originals support to MM tables (and make the conversion a metadata only > operation) > -- > > Key: HIVE-19258 > URL: https://issues.apache.org/jira/browse/HIVE-19258 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19258.01.patch, HIVE-19258.02.patch, > HIVE-19258.03.patch, HIVE-19258.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19448) Vectorization: sysdb test doesn't work after enabling vectorization by default
[
https://issues.apache.org/jira/browse/HIVE-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline reassigned HIVE-19448:
---
> Vectorization: sysdb test doesn't work after enabling vectorization by default
> --
>
> Key: HIVE-19448
> URL: https://issues.apache.org/jira/browse/HIVE-19448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>
> {noformat}
> Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
> java.lang.Boolean
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaBooleanObjectInspector.getPrimitiveWritableObject(JavaBooleanObjectInspector.java:36)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:434)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:347)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:948){noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18126) IOW Mechanics of multiple commands with OVERWRITE in a singe transactions
[ https://issues.apache.org/jira/browse/HIVE-18126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18126: -- Target Version/s: 3.0.0 Priority: Critical (was: Major) > IOW Mechanics of multiple commands with OVERWRITE in a singe transactions > - > > Key: HIVE-18126 > URL: https://issues.apache.org/jira/browse/HIVE-18126 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > > for Insert overwrite/load data overwrite we create base_x/ to hold the data > thus are able to make Overwrite command non-blocking. > What happens if multiple IOWs are run against the same table/partition in the > same transaction. > should base support a suffix base_x_000 like deltas? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-18380) ALTER TABLE CONCATENATE is not supported on Micro-managed table
[ https://issues.apache.org/jira/browse/HIVE-18380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-18380. --- Resolution: Fixed Fix Version/s: 3.0.0 Release Note: n/a fixed as part of HIVE-18288 > ALTER TABLE CONCATENATE is not supported on Micro-managed table > --- > > Key: HIVE-18380 > URL: https://issues.apache.org/jira/browse/HIVE-18380 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Eugene Koifman >Priority: Minor > Fix For: 3.0.0 > > > ALTER TABLE CONCATENATE is not supported on Micro-managed table. > Example qtest is "alter_merge_2_orc.q" and the unsupportability is revealed > when we start with tables in micro-managed table type(insert_only > transactional). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19332) Disable compute.query.using.stats for external table
[
https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1642#comment-1642
]
Hive QA commented on HIVE-19332:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922344/HIVE-19332.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 220 failed/errored test(s), 13539 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Updated] (HIVE-18774) ACID: Use the _copy_N files copyNumber as the implicit statement-id
[ https://issues.apache.org/jira/browse/HIVE-18774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18774: -- Target Version/s: 3.0.0 Environment: if this is not done in 3.0 it cannot be done at all > ACID: Use the _copy_N files copyNumber as the implicit statement-id > --- > > Key: HIVE-18774 > URL: https://issues.apache.org/jira/browse/HIVE-18774 > Project: Hive > Issue Type: Bug > Components: Transactions > Environment: if this is not done in 3.0 it cannot be done at all >Reporter: Gopal V >Assignee: Eugene Koifman >Priority: Blocker > > When upgrading flat ORC files to ACID, use the _copy_N numbering as a > statement-id to avoid having to align the row numbering between _copy_1 and > _copy_2 files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18709) Enable Compaction to work on more than one partition per job
[ https://issues.apache.org/jira/browse/HIVE-18709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1640#comment-1640 ] Eugene Koifman commented on HIVE-18709: --- with query based compactor this may be easy > Enable Compaction to work on more than one partition per job > > > Key: HIVE-18709 > URL: https://issues.apache.org/jira/browse/HIVE-18709 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > currently compaction launches 1 MR job per partition that needs to be > compacted. > The number of tasks is equal to the number of buckets in the table (or number > or writers in the 'widest' write). > The number of AMs in a cluster is usually limited to a small percentage of > the nodes. This limits how much compaction can be done in parallel. > Investigate what it would take for a single job to be able to handle multiple > partitions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18774) ACID: Use the _copy_N files copyNumber as the implicit statement-id
[ https://issues.apache.org/jira/browse/HIVE-18774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18774: -- Priority: Blocker (was: Major) > ACID: Use the _copy_N files copyNumber as the implicit statement-id > --- > > Key: HIVE-18774 > URL: https://issues.apache.org/jira/browse/HIVE-18774 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Gopal V >Assignee: Eugene Koifman >Priority: Blocker > > When upgrading flat ORC files to ACID, use the _copy_N numbering as a > statement-id to avoid having to align the row numbering between _copy_1 and > _copy_2 files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18748) Rename table impacts the ACID behaviour as table names are not updated in meta-tables.
[ https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466657#comment-16466657 ] Eugene Koifman commented on HIVE-18748: --- [~sankarh] could you review please > Rename table impacts the ACID behaviour as table names are not updated in > meta-tables. > -- > > Key: HIVE-18748 > URL: https://issues.apache.org/jira/browse/HIVE-18748 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Eugene Koifman >Priority: Critical > Labels: ACID, DDL > Fix For: 3.0.0 > > Attachments: HIVE-18748.02.patch > > > ACID implementation uses metatables such as TXN_COMPONENTS, > COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to > manage ACID operations. > Per table write ID implementation (HIVE-18192) introduces couple of > metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids > allocated per table. > Now, when we rename any tables, it is necessary to update the corresponding > table names in these metatables as well. Otherwise, ACID table operations > won't work properly. > Since, this change is significant and have other side-effects, we propose to > disable rename tables on ACID tables until a fix is figured out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466648#comment-16466648
]
Sahil Takiar commented on HIVE-18533:
-
[~lirui] yeah that might be slightly simpler. Although the specific example you
mentioned shouldn't be an issue. Calling {{SparkAppHandle#stop}} should cause
the Spark App to transition to a stopped state, in which case the
{{SparkAppHandle.Listener}} will decrement the count down latch.
I don't think it would be as simple as wrapping the countdown latch in a
{{FutureTask}}, there still needs to be a way to cancel the underlying Spark
app if {{#interrupt}} is called.
Plus there are unit tests for {{SparkLauncherFuture}}, although I can add in
some more to ensure the {{Future}} contract isn't broken.
> Add option to use InProcessLauncher to submit spark jobs
>
>
> Key: HIVE-18533
> URL: https://issues.apache.org/jira/browse/HIVE-18533
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18533.1.patch, HIVE-18533.2.patch,
> HIVE-18533.3.patch, HIVE-18533.4.patch, HIVE-18533.5.patch,
> HIVE-18533.6.patch, HIVE-18533.7.patch, HIVE-18533.8.patch,
> HIVE-18533.9.patch, HIVE-18533.91.patch, HIVE-18831.93.patch
>
>
> See discussion in HIVE-16484 for details.
> I think this will help with reducing the amount of time it takes to open a
> HoS session + debuggability (no need launch a separate process to run a Spark
> app).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18748) Rename table impacts the ACID behaviour as table names are not updated in meta-tables.
[ https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18748: -- Attachment: HIVE-18748.02.patch > Rename table impacts the ACID behaviour as table names are not updated in > meta-tables. > -- > > Key: HIVE-18748 > URL: https://issues.apache.org/jira/browse/HIVE-18748 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Eugene Koifman >Priority: Critical > Labels: ACID, DDL > Fix For: 3.0.0 > > Attachments: HIVE-18748.02.patch > > > ACID implementation uses metatables such as TXN_COMPONENTS, > COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to > manage ACID operations. > Per table write ID implementation (HIVE-18192) introduces couple of > metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids > allocated per table. > Now, when we rename any tables, it is necessary to update the corresponding > table names in these metatables as well. Otherwise, ACID table operations > won't work properly. > Since, this change is significant and have other side-effects, we propose to > disable rename tables on ACID tables until a fix is figured out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18748) Rename table impacts the ACID behaviour as table names are not updated in meta-tables.
[ https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18748: -- Status: Patch Available (was: Open) > Rename table impacts the ACID behaviour as table names are not updated in > meta-tables. > -- > > Key: HIVE-18748 > URL: https://issues.apache.org/jira/browse/HIVE-18748 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Eugene Koifman >Priority: Critical > Labels: ACID, DDL > Fix For: 3.0.0 > > Attachments: HIVE-18748.02.patch > > > ACID implementation uses metatables such as TXN_COMPONENTS, > COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to > manage ACID operations. > Per table write ID implementation (HIVE-18192) introduces couple of > metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids > allocated per table. > Now, when we rename any tables, it is necessary to update the corresponding > table names in these metatables as well. Otherwise, ACID table operations > won't work properly. > Since, this change is significant and have other side-effects, we propose to > disable rename tables on ACID tables until a fix is figured out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19307 started by Eric Wohlstadter. --- > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Status: Open (was: Patch Available) > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Attachment: HIVE-19307.4.patch > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19332) Disable compute.query.using.stats for external table
[
https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466615#comment-16466615
]
Hive QA commented on HIVE-19332:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 2323 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
51s{color} | {color:blue} standalone-metastore in master has 215 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
54s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 4 new + 548 unchanged - 0
fixed = 552 total (was 548) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10753/dev-support/hive-personality.sh
|
| git revision | master / 0930aec |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10753/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10753/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10753/yetus/patch-asflicense-problems.txt
|
| modules | C: itests ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10753/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Disable compute.query.using.stats for external table
>
>
> Key: HIVE-19332
> URL: https://issues.apache.org/jira/browse/HIVE-19332
> Project: Hive
> Issue Type: Sub-task
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-19332.1.patch, HIVE-19332.2.patch,
> HIVE-19332.3.patch, HIVE-19332.4.patch
>
>
> Hive can use statistics to answer queries like count(*). This can be
> problematic on external tables where another tool might add files that Hive
> doesn’t know about. In that case Hive will return incorrect results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19433) HiveJoinPushTransitivePredicatesRule hangs
[
https://issues.apache.org/jira/browse/HIVE-19433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466592#comment-16466592
]
Vineet Garg commented on HIVE-19433:
[~jcamachorodriguez] Can you take a look?
Issue was with the way transitive predicate logic permutes over input
references of a predicate (to infer new predicates).
> HiveJoinPushTransitivePredicatesRule hangs
> --
>
> Key: HIVE-19433
> URL: https://issues.apache.org/jira/browse/HIVE-19433
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19433.1.patch
>
>
> *Reproducer*
> {code:sql}
> CREATE TABLE `table1`(
>`idp_warehouse_id` bigint,
>`idp_audit_id` bigint,
>`idp_effective_date` date,
>`idp_end_date` date,
>`idp_delete_date` date,
>`pruid` varchar(32),
>`prid` bigint,
>`prtimesheetid` bigint,
>`prassignmentid` bigint,
>`prchargecodeid` bigint,
>`prtypecodeid` bigint,
>`prsequence` bigint,
>`prmodby` varchar(96),
>`prmodtime` timestamp,
>`prrmexported` bigint,
>`prrmckdel` bigint,
>`slice_status` int,
>`role_id` bigint,
>`user_lov1` varchar(30),
>`user_lov2` varchar(30),
>`incident_id` bigint,
>`incident_investment_id` bigint,
>`odf_ss_actuals` bigint,
>`practsum` decimal(38,20));
> CREATE TABLE `table2`(
>`idp_warehouse_id` bigint,
>`idp_audit_id` bigint,
>`idp_effective_date` date,
>`idp_end_date` date,
>`idp_delete_date` date,
>`pruid` varchar(32),
>`prid` bigint,
>`prtimesheetid` bigint,
>`prassignmentid` bigint,
>`prchargecodeid` bigint,
>`prtypecodeid` bigint,
>`prsequence` bigint,
>`prmodby` varchar(96),
>`prmodtime` timestamp,
>`prrmexported` bigint,
>`prrmckdel` bigint,
>`slice_status` int,
>`role_id` bigint,
>`user_lov1` varchar(30),
>`user_lov2` varchar(30),
>`incident_id` bigint,
>`incident_investment_id` bigint,
>`odf_ss_actuals` bigint,
>`practsum` decimal(38,20));
> explain SELECT s.idp_warehouse_id AS source_warehouse_id
> FROMtable1 s
> JOIN
>table2 d
> ON (
> s.prid = d.prid )
> JOIN
> table2 e
> ON
> s.prid = e.prid
> WHERE
> concat(
> CASE
> WHEN s.prid IS NULL THEN 1
> ELSE s.prid
> END,',',
> CASE
> WHEN s.prtimesheetid IS NULL THEN 1
> ELSE s.prtimesheetid
> END,',',
> CASE
> WHEN s.prassignmentid IS NULL THEN 1
> ELSE s.prassignmentid
> END,',',
> CASE
> WHEN s.prchargecodeid IS NULL THEN 1
> ELSE s.prchargecodeid
> END,',',
> CASE
> WHEN (s.prtypecodeid) IS NULL THEN ''
> ELSE s.prtypecodeid
> END,',',
> CASE
> WHEN s.practsum IS NULL THEN 1
> ELSE s.practsum
> END,',',
> CASE
> WHEN s.prsequence IS NULL THEN 1
> ELSE s.prsequence
> END,',',
> CASE
> WHEN length(s.prmodby) IS NULL THEN ''
> ELSE s.prmodby
> END,',',
> CASE
> WHEN s.prmodtime IS NULL THEN
> cast(from_unixtime(unix_timestamp('2017-12-08','-MM-dd') ) AS timestamp)
> ELSE s.prmodtime
> END,',',
> CASE
> WHEN s.prrmexported IS NULL THEN 1
> ELSE s.prrmexported
> END,',',
> CASE
> WHEN s.prrmckdel IS NULL THEN 1
> ELSE s.prrmckdel
> END,',',
>
[jira] [Commented] (HIVE-14388) Add number of rows inserted message after insert command in Beeline
[ https://issues.apache.org/jira/browse/HIVE-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466585#comment-16466585 ] Bharathkrishna Guruvayoor Murali commented on HIVE-14388: - Test failures fixed. Current failures are unrelated. > Add number of rows inserted message after insert command in Beeline > --- > > Key: HIVE-14388 > URL: https://issues.apache.org/jira/browse/HIVE-14388 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vihang Karajgaonkar >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Minor > Attachments: HIVE-14388-WIP.patch, HIVE-14388.02.patch, > HIVE-14388.03.patch, HIVE-14388.05.patch, HIVE-14388.06.patch, > HIVE-14388.07.patch, HIVE-14388.08.patch > > > Currently, when you run insert command on beeline, it returns a message > saying "No rows affected .." > A better and more intuitive msg would be "xxx rows inserted (26.068 seconds)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19446) QueryCache: Transaction lists needed for pending cache entries
[ https://issues.apache.org/jira/browse/HIVE-19446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466582#comment-16466582 ] Vineet Garg commented on HIVE-19446: +1 for branch-3 > QueryCache: Transaction lists needed for pending cache entries > -- > > Key: HIVE-19446 > URL: https://issues.apache.org/jira/browse/HIVE-19446 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19446.1.patch > > > Hive query-cache needs a transactional list, even when the entry is pending > state so that other identical queries with the same transactional state can > wait for the first query to complete, instead of triggering their own > instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19381) Function replication in cloud fail when download resource from AWS
[
https://issues.apache.org/jira/browse/HIVE-19381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466580#comment-16466580
]
Hive QA commented on HIVE-19381:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921480/HIVE-19381.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10752/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10752/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10752/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-05-07 22:49:45.917
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-10752/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-05-07 22:49:45.921
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
7ebcdeb..0930aec master -> origin/master
d81d17f..8d1f636 branch-2 -> origin/branch-2
* [new branch] storage-branch-2.6.1 -> origin/storage-branch-2.6.1
* [new tag] storage-release-2.6.1-rc0 -> storage-release-2.6.1-rc0
+ git reset --hard HEAD
HEAD is now at 7ebcdeb HIVE-17657 : export/import for MM tables is broken
(Sergey Shelukhin, reviewed by Eugene Koifman)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 0930aec HIVE-19312 : MM tables don't work with BucketizedHIF
(Sergey Shelukhin, reviewed by Gunther Hagleitner)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-05-07 22:49:53.942
+ rm -rf ../yetus_PreCommit-HIVE-Build-10752
+ mkdir ../yetus_PreCommit-HIVE-Build-10752
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-10752
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-10752/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java: does not
exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc2493921929003840944.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc2493921929003840944.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
log4j:WARN No appenders could be found for logger (DataNucleus.Persistence).
log4j:WARN Please initialize the log4j system properly.
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer completed with success for 40 classes.
ANTLR Parser Generator Version 3.5.2
Output file
[jira] [Commented] (HIVE-14388) Add number of rows inserted message after insert command in Beeline
[
https://issues.apache.org/jira/browse/HIVE-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466574#comment-16466574
]
Hive QA commented on HIVE-14388:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922304/HIVE-14388.08.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 13537 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-19446) QueryCache: Transaction lists needed for pending cache entries
[ https://issues.apache.org/jira/browse/HIVE-19446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466567#comment-16466567 ] Jason Dere commented on HIVE-19446: --- [~vgarg] can this one go into branch-3? > QueryCache: Transaction lists needed for pending cache entries > -- > > Key: HIVE-19446 > URL: https://issues.apache.org/jira/browse/HIVE-19446 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19446.1.patch > > > Hive query-cache needs a transactional list, even when the entry is pending > state so that other identical queries with the same transactional state can > wait for the first query to complete, instead of triggering their own > instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18052) Run p-tests on mm tables
[ https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466566#comment-16466566 ] Sergey Shelukhin edited comment on HIVE-18052 at 5/7/18 10:45 PM: -- After the recent fixes and changes to MM tables, we can do another iteration. Most of the changes above are irrelevant, or have since been fixed. One major item remaining is stats for ACID and MM tables; that is tracked separately. I also removed the out file changes from .19 patch from this iteration for now. They create confusing conflicts and we don't know if out file diffs are still the old irrelevant diffs, or there are some new relevant diffs after so many changes to MM tables and to the tests themselves. was (Author: sershe): After the recent fixes and changes to MM tables, we can do another iteration. I also removed the out file changes from .19 patch from this iteration for now. > Run p-tests on mm tables > > > Key: HIVE-18052 > URL: https://issues.apache.org/jira/browse/HIVE-18052 > Project: Hive > Issue Type: Task >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18052.1.patch, HIVE-18052.10.patch, > HIVE-18052.11.patch, HIVE-18052.12.patch, HIVE-18052.13.patch, > HIVE-18052.14.patch, HIVE-18052.15.patch, HIVE-18052.16.patch, > HIVE-18052.17.patch, HIVE-18052.18.patch, HIVE-18052.19.patch, > HIVE-18052.2.patch, HIVE-18052.20.patch, HIVE-18052.3.patch, > HIVE-18052.4.patch, HIVE-18052.5.patch, HIVE-18052.6.patch, > HIVE-18052.7.patch, HIVE-18052.8.patch, HIVE-18052.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18052) Run p-tests on mm tables
[ https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18052: Attachment: HIVE-18052.20.patch > Run p-tests on mm tables > > > Key: HIVE-18052 > URL: https://issues.apache.org/jira/browse/HIVE-18052 > Project: Hive > Issue Type: Task >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18052.1.patch, HIVE-18052.10.patch, > HIVE-18052.11.patch, HIVE-18052.12.patch, HIVE-18052.13.patch, > HIVE-18052.14.patch, HIVE-18052.15.patch, HIVE-18052.16.patch, > HIVE-18052.17.patch, HIVE-18052.18.patch, HIVE-18052.19.patch, > HIVE-18052.2.patch, HIVE-18052.20.patch, HIVE-18052.3.patch, > HIVE-18052.4.patch, HIVE-18052.5.patch, HIVE-18052.6.patch, > HIVE-18052.7.patch, HIVE-18052.8.patch, HIVE-18052.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18052) Run p-tests on mm tables
[ https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466566#comment-16466566 ] Sergey Shelukhin commented on HIVE-18052: - After the recent fixes and changes to MM tables, we can do another iteration. I also removed the out file changes from .19 patch from this iteration for now. > Run p-tests on mm tables > > > Key: HIVE-18052 > URL: https://issues.apache.org/jira/browse/HIVE-18052 > Project: Hive > Issue Type: Task >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18052.1.patch, HIVE-18052.10.patch, > HIVE-18052.11.patch, HIVE-18052.12.patch, HIVE-18052.13.patch, > HIVE-18052.14.patch, HIVE-18052.15.patch, HIVE-18052.16.patch, > HIVE-18052.17.patch, HIVE-18052.18.patch, HIVE-18052.19.patch, > HIVE-18052.2.patch, HIVE-18052.20.patch, HIVE-18052.3.patch, > HIVE-18052.4.patch, HIVE-18052.5.patch, HIVE-18052.6.patch, > HIVE-18052.7.patch, HIVE-18052.8.patch, HIVE-18052.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19446) QueryCache: Transaction lists needed for pending cache entries
[ https://issues.apache.org/jira/browse/HIVE-19446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19446: -- Status: Patch Available (was: Open) > QueryCache: Transaction lists needed for pending cache entries > -- > > Key: HIVE-19446 > URL: https://issues.apache.org/jira/browse/HIVE-19446 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19446.1.patch > > > Hive query-cache needs a transactional list, even when the entry is pending > state so that other identical queries with the same transactional state can > wait for the first query to complete, instead of triggering their own > instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19446) QueryCache: Transaction lists needed for pending cache entries
[ https://issues.apache.org/jira/browse/HIVE-19446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19446: -- Attachment: HIVE-19446.1.patch > QueryCache: Transaction lists needed for pending cache entries > -- > > Key: HIVE-19446 > URL: https://issues.apache.org/jira/browse/HIVE-19446 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19446.1.patch > > > Hive query-cache needs a transactional list, even when the entry is pending > state so that other identical queries with the same transactional state can > wait for the first query to complete, instead of triggering their own > instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19447) BucketizedHiveInputFormat doesn't account for ACID
[ https://issues.apache.org/jira/browse/HIVE-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19447: -- Component/s: Transactions > BucketizedHiveInputFormat doesn't account for ACID > -- > > Key: HIVE-19447 > URL: https://issues.apache.org/jira/browse/HIVE-19447 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Priority: Major > > See the TODO added in HIVE-19312. > It doesn't account for MM tables and can apparently be used for them, > producing incorrect results. > I'm not sure if it can be used for ACID tables; we need to fix it for ACID > (w.r.t. the ACID-related logic in HIF) or perhaps add a negative test where > for the same query it's used for a non-ACID table but not used for an ACID > table. mm_bhif test has a simple example query (count distinct iirc) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19447) BucketizedHiveInputFormat doesn't account for ACID
[ https://issues.apache.org/jira/browse/HIVE-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466556#comment-16466556 ] Sergey Shelukhin commented on HIVE-19447: - cc [~ekoifman] > BucketizedHiveInputFormat doesn't account for ACID > -- > > Key: HIVE-19447 > URL: https://issues.apache.org/jira/browse/HIVE-19447 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > See the TODO added in HIVE-19312. > It doesn't account for MM tables and can apparently be used for them, > producing incorrect results. > I'm not sure if it can be used for ACID tables; we need to fix it for ACID > (w.r.t. the ACID-related logic in HIF) or perhaps add a negative test where > for the same query it's used for a non-ACID table but not used for an ACID > table. mm_bhif test has a simple example query (count distinct iirc) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19446) QueryCache: Transaction lists needed for pending cache entries
[ https://issues.apache.org/jira/browse/HIVE-19446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-19446: - Assignee: Jason Dere > QueryCache: Transaction lists needed for pending cache entries > -- > > Key: HIVE-19446 > URL: https://issues.apache.org/jira/browse/HIVE-19446 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Jason Dere >Priority: Major > > Hive query-cache needs a transactional list, even when the entry is pending > state so that other identical queries with the same transactional state can > wait for the first query to complete, instead of triggering their own > instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19446) QueryCache: Transaction lists needed for pending cache entries
[
https://issues.apache.org/jira/browse/HIVE-19446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466555#comment-16466555
]
Gopal V commented on HIVE-19446:
Lookups that happen too early are triggering this NPE.
{code}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.entryMatches(QueryResultsCache.java:705)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.lookup(QueryResultsCache.java:442)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.checkResultsCache(SemanticAnalyzer.java:14703)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12060)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:334)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:288)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
Show more
{code}
> QueryCache: Transaction lists needed for pending cache entries
> --
>
> Key: HIVE-19446
> URL: https://issues.apache.org/jira/browse/HIVE-19446
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Jason Dere
>Priority: Major
>
> Hive query-cache needs a transactional list, even when the entry is pending
> state so that other identical queries with the same transactional state can
> wait for the first query to complete, instead of triggering their own
> instance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19381) Function replication in cloud fail when download resource from AWS
[ https://issues.apache.org/jira/browse/HIVE-19381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466554#comment-16466554 ] Daniel Dai commented on HIVE-19381: --- Yes, this needs to go into 3.0 release. Always get lost in ptest. Kicking off again. > Function replication in cloud fail when download resource from AWS > -- > > Key: HIVE-19381 > URL: https://issues.apache.org/jira/browse/HIVE-19381 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19381.1.patch > > > Another case replication shall use the config in with clause. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19312) MM tables don't work with BucketizedHIF
[ https://issues.apache.org/jira/browse/HIVE-19312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19312: Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Committed to master after running some tests locally. Thanks for the review! > MM tables don't work with BucketizedHIF > --- > > Key: HIVE-19312 > URL: https://issues.apache.org/jira/browse/HIVE-19312 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19312.01.patch, HIVE-19312.02.patch, > HIVE-19312.03.patch, HIVE-19312.04.patch, HIVE-19312.05.patch, > HIVE-19312.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14388) Add number of rows inserted message after insert command in Beeline
[
https://issues.apache.org/jira/browse/HIVE-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466531#comment-16466531
]
Hive QA commented on HIVE-14388:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} beeline in master has 69 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} jdbc in master has 17 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} service in master has 50 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} The patch service-rpc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch beeline passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} jdbc: The patch generated 0 new + 4 unchanged - 2
fixed = 4 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} ql: The patch generated 0 new + 217 unchanged - 13
fixed = 217 total (was 230) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch service passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10749/dev-support/hive-personality.sh
|
| git revision | master / d159f24 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
[jira] [Commented] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[
https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466520#comment-16466520
]
Eric Wohlstadter commented on HIVE-19307:
-
[~jdere]
I created HIVE-19445 to followup on:
{quote}I see now that the writeListener could be implemented in such a way as
to propagate a write error back to the writer (so we can possibly throw an
exception and fail the current operation rather than just log and ignore the
error). Plus on close I'm wondering if it is better just to wait for the close
future to complete so we can check the status.
{quote}
Will upload a new patch to fix the other suggestions.
> Support ArrowOutputStream in LlapOutputFormatService
>
>
> Key: HIVE-19307
> URL: https://issues.apache.org/jira/browse/HIVE-19307
> Project: Hive
> Issue Type: Task
> Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19307.3.patch
>
>
> Support pushing arrow batches through
> org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19445) Graceful handling of "close" in WritableByteChannelAdapter
[
https://issues.apache.org/jira/browse/HIVE-19445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19445:
Issue Type: Sub-task (was: Bug)
Parent: HIVE-19305
> Graceful handling of "close" in WritableByteChannelAdapter
> --
>
> Key: HIVE-19445
> URL: https://issues.apache.org/jira/browse/HIVE-19445
> Project: Hive
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Priority: Major
>
> org.apache.hadoop.hive.llap.WritableByteChannelAdapter
> {quote}"I see now that the writeListener could be implemented in such a way
> as to propagate a write error back to the writer (so we can possibly throw an
> exception and fail the current operation rather than just log and ignore the
> error). Plus on close I'm wondering if it is better just to wait for the
> close future to complete so we can check the status."
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields
[ https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466509#comment-16466509 ] Vihang Karajgaonkar commented on HIVE-19041: Tests failures are unrelated. > Thrift deserialization of Partition objects should intern fields > > > Key: HIVE-19041 > URL: https://issues.apache.org/jira/browse/HIVE-19041 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0, 2.3.2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-19041.01.patch, HIVE-19041.02.patch, > HIVE-19041.03.patch > > > When a client is creating large number of partitions, the thrift objects are > deserialized into Partition objects. The read method of these objects does > not intern the inputformat, location, outputformat which cause large number > of duplicate Strings in the HMS memory. We should intern these objects while > deserialization to reduce memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19334) Use actual file size rather than stats for fetch task optimization with external tables
[ https://issues.apache.org/jira/browse/HIVE-19334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19334: -- Attachment: HIVE-19334.3.patch > Use actual file size rather than stats for fetch task optimization with > external tables > --- > > Key: HIVE-19334 > URL: https://issues.apache.org/jira/browse/HIVE-19334 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19334.1.patch, HIVE-19334.2.patch, > HIVE-19334.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19336) Disable SMB/Bucketmap join for external tables
[ https://issues.apache.org/jira/browse/HIVE-19336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466506#comment-16466506 ] Jason Dere commented on HIVE-19336: --- PTest never ran - attaching same patch again. > Disable SMB/Bucketmap join for external tables > -- > > Key: HIVE-19336 > URL: https://issues.apache.org/jira/browse/HIVE-19336 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19336.1.patch, HIVE-19336.2.patch, > HIVE-19336.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19336) Disable SMB/Bucketmap join for external tables
[ https://issues.apache.org/jira/browse/HIVE-19336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19336: -- Attachment: HIVE-19336.3.patch > Disable SMB/Bucketmap join for external tables > -- > > Key: HIVE-19336 > URL: https://issues.apache.org/jira/browse/HIVE-19336 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19336.1.patch, HIVE-19336.2.patch, > HIVE-19336.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19332) Disable compute.query.using.stats for external table
[ https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466504#comment-16466504 ] Jason Dere commented on HIVE-19332: --- Didn't get a chance to look at the precommit results - re-attaching same patch. > Disable compute.query.using.stats for external table > > > Key: HIVE-19332 > URL: https://issues.apache.org/jira/browse/HIVE-19332 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19332.1.patch, HIVE-19332.2.patch, > HIVE-19332.3.patch, HIVE-19332.4.patch > > > Hive can use statistics to answer queries like count(*). This can be > problematic on external tables where another tool might add files that Hive > doesn’t know about. In that case Hive will return incorrect results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19332) Disable compute.query.using.stats for external table
[ https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19332: -- Attachment: HIVE-19332.4.patch > Disable compute.query.using.stats for external table > > > Key: HIVE-19332 > URL: https://issues.apache.org/jira/browse/HIVE-19332 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19332.1.patch, HIVE-19332.2.patch, > HIVE-19332.3.patch, HIVE-19332.4.patch > > > Hive can use statistics to answer queries like count(*). This can be > problematic on external tables where another tool might add files that Hive > doesn’t know about. In that case Hive will return incorrect results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19388) ClassCastException during VectorMapJoinCommonOperator initialization
[
https://issues.apache.org/jira/browse/HIVE-19388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-19388:
---
Fix Version/s: 2.4.0
> ClassCastException during VectorMapJoinCommonOperator initialization
>
>
> Key: HIVE-19388
> URL: https://issues.apache.org/jira/browse/HIVE-19388
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2, 3.1.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 2.4.0, 3.1.0
>
> Attachments: HIVE-19388.01.patch, HIVE-19388.02.patch
>
>
> I see the following exceptions when I a mapjoin operator is being initialized
> on Hive-on-Spark and when vectorization is turned on.
> This happens when the hashTable is empty. The code in
> {{MapJoinTableContainerSerDe#getDefaultEmptyContainer}} method returns a
> HashMapWrapper while the VectorMapJoinOperator expects a
> {{MapJoinBytesTableContainer}} when {{hive.mapjoin.optimized.hashtable}} is
> set to true.
> {noformat}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper cannot be cast to
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerDirectAccess
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedHashTable.(VectorMapJoinOptimizedHashTable.java:92)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedHashMap.(VectorMapJoinOptimizedHashMap.java:127)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedStringHashMap.(VectorMapJoinOptimizedStringHashMap.java:60)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedCreateHashTable.createHashTable(VectorMapJoinOptimizedCreateHashTable.java:80)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.setUpHashTable(VectorMapJoinCommonOperator.java:485)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.completeInitializationOp(VectorMapJoinCommonOperator.java:461)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:471)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:401)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:574)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:526)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:387)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:109)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19388) ClassCastException during VectorMapJoinCommonOperator initialization
[
https://issues.apache.org/jira/browse/HIVE-19388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466494#comment-16466494
]
Vihang Karajgaonkar commented on HIVE-19388:
Thanks for the review [~lirui]. Patch merged to branch-2 and master
> ClassCastException during VectorMapJoinCommonOperator initialization
>
>
> Key: HIVE-19388
> URL: https://issues.apache.org/jira/browse/HIVE-19388
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2, 3.1.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 2.4.0, 3.1.0
>
> Attachments: HIVE-19388.01.patch, HIVE-19388.02.patch
>
>
> I see the following exceptions when I a mapjoin operator is being initialized
> on Hive-on-Spark and when vectorization is turned on.
> This happens when the hashTable is empty. The code in
> {{MapJoinTableContainerSerDe#getDefaultEmptyContainer}} method returns a
> HashMapWrapper while the VectorMapJoinOperator expects a
> {{MapJoinBytesTableContainer}} when {{hive.mapjoin.optimized.hashtable}} is
> set to true.
> {noformat}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper cannot be cast to
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerDirectAccess
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedHashTable.(VectorMapJoinOptimizedHashTable.java:92)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedHashMap.(VectorMapJoinOptimizedHashMap.java:127)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedStringHashMap.(VectorMapJoinOptimizedStringHashMap.java:60)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedCreateHashTable.createHashTable(VectorMapJoinOptimizedCreateHashTable.java:80)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.setUpHashTable(VectorMapJoinCommonOperator.java:485)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.completeInitializationOp(VectorMapJoinCommonOperator.java:461)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:471)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:401)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:574)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:526)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:387)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:109)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[ https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko updated HIVE-12342: - Attachment: HIVE-12342.09.patch > Set default value of hive.optimize.index.filter to true > --- > > Key: HIVE-12342 > URL: https://issues.apache.org/jira/browse/HIVE-12342 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-12342.05.patch, HIVE-12342.06.patch, > HIVE-12342.07.patch, HIVE-12342.08.patch, HIVE-12342.09.patch, > HIVE-12342.1.patch, HIVE-12342.2.patch, HIVE-12342.3.patch, > HIVE-12342.4.patch, HIVE-12342.patch > > > This configuration governs ppd for storage layer. When applicable, it will > always help. It should be on by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19388) ClassCastException during VectorMapJoinCommonOperator initialization
[
https://issues.apache.org/jira/browse/HIVE-19388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-19388:
---
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
> ClassCastException during VectorMapJoinCommonOperator initialization
>
>
> Key: HIVE-19388
> URL: https://issues.apache.org/jira/browse/HIVE-19388
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2, 3.1.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19388.01.patch, HIVE-19388.02.patch
>
>
> I see the following exceptions when I a mapjoin operator is being initialized
> on Hive-on-Spark and when vectorization is turned on.
> This happens when the hashTable is empty. The code in
> {{MapJoinTableContainerSerDe#getDefaultEmptyContainer}} method returns a
> HashMapWrapper while the VectorMapJoinOperator expects a
> {{MapJoinBytesTableContainer}} when {{hive.mapjoin.optimized.hashtable}} is
> set to true.
> {noformat}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper cannot be cast to
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerDirectAccess
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedHashTable.(VectorMapJoinOptimizedHashTable.java:92)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedHashMap.(VectorMapJoinOptimizedHashMap.java:127)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedStringHashMap.(VectorMapJoinOptimizedStringHashMap.java:60)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedCreateHashTable.createHashTable(VectorMapJoinOptimizedCreateHashTable.java:80)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.setUpHashTable(VectorMapJoinCommonOperator.java:485)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.completeInitializationOp(VectorMapJoinCommonOperator.java:461)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:471)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:401)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:574)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:526)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:387)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:109)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19310) Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
[
https://issues.apache.org/jira/browse/HIVE-19310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466457#comment-16466457
]
Sergey Shelukhin commented on HIVE-19310:
-
[~vihangk1] I saw your name on top of the 3.0 thread on dev@ and assumed you
are the RM :)
> Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might
> need to be run only in test env
> -
>
> Key: HIVE-19310
> URL: https://issues.apache.org/jira/browse/HIVE-19310
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19310.02.patch, HIVE-19310.03.patch,
> HIVE-19310.1.patch
>
>
> MetaStoreDirectSql.ensureDbInit has the following 2 calls which we have
> observed taking a long time in our testing:
> {code}
> initQueries.add(pm.newQuery(MNotificationLog.class, "dbName == ''"));
> initQueries.add(pm.newQuery(MNotificationNextId.class, "nextEventId < -1"));
> {code}
> In a production environment, these tables should be initialized using
> schematool, however in a test environment, these calls might be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19310) Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
[
https://issues.apache.org/jira/browse/HIVE-19310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466453#comment-16466453
]
Vihang Karajgaonkar commented on HIVE-19310:
guess you meant Vineet [~sershe] :)
> Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might
> need to be run only in test env
> -
>
> Key: HIVE-19310
> URL: https://issues.apache.org/jira/browse/HIVE-19310
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19310.02.patch, HIVE-19310.03.patch,
> HIVE-19310.1.patch
>
>
> MetaStoreDirectSql.ensureDbInit has the following 2 calls which we have
> observed taking a long time in our testing:
> {code}
> initQueries.add(pm.newQuery(MNotificationLog.class, "dbName == ''"));
> initQueries.add(pm.newQuery(MNotificationNextId.class, "nextEventId < -1"));
> {code}
> In a production environment, these tables should be initialized using
> schematool, however in a test environment, these calls might be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19310) Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
[
https://issues.apache.org/jira/browse/HIVE-19310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19310:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Committed also to branch-3. Thanks!
> Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might
> need to be run only in test env
> -
>
> Key: HIVE-19310
> URL: https://issues.apache.org/jira/browse/HIVE-19310
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19310.02.patch, HIVE-19310.03.patch,
> HIVE-19310.1.patch
>
>
> MetaStoreDirectSql.ensureDbInit has the following 2 calls which we have
> observed taking a long time in our testing:
> {code}
> initQueries.add(pm.newQuery(MNotificationLog.class, "dbName == ''"));
> initQueries.add(pm.newQuery(MNotificationNextId.class, "nextEventId < -1"));
> {code}
> In a production environment, these tables should be initialized using
> schematool, however in a test environment, these calls might be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17657) export/import for MM tables is broken
[ https://issues.apache.org/jira/browse/HIVE-17657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17657: Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Rebased yet again, ran some tests locally and committed to master. HiveQA is impossible to use. > export/import for MM tables is broken > - > > Key: HIVE-17657 > URL: https://issues.apache.org/jira/browse/HIVE-17657 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Fix For: 3.1.0 > > Attachments: HIVE-17657.01.patch, HIVE-17657.02.patch, > HIVE-17657.03.patch, HIVE-17657.04.patch, HIVE-17657.05.patch, > HIVE-17657.06.patch, HIVE-17657.07.patch, HIVE-17657.08.patch, > HIVE-17657.09.patch, HIVE-17657.10.patch, HIVE-17657.patch > > > there is mm_exim.q but it's not clear from the tests what file structure it > creates > On import the txnids in the directory names would have to be remapped if > importing to a different cluster. Perhaps export can be smart and export > highest base_x and accretive deltas (minus aborted ones). Then import can > ...? It would have to remap txn ids from the archive to new txn ids. This > would then mean that import is made up of several transactions rather than 1 > atomic op. (all locks must belong to a transaction) > One possibility is to open a new txn for each dir in the archive (where > start/end txn of file name is the same) and commit all of them at once (need > new TMgr API for that). This assumes using a shared lock (if any!) and thus > allows other inserts (not related to import) to occur. > What if you have delta_6_9, such as a result of concatenate? If we stipulate > that this must mean that there is no delta_6_6 or any other "obsolete" delta > in the archive we can map it to a new single txn delta_x_x. > Add read_only mode for tables (useful in general, may be needed for upgrade > etc) and use that to make the above atomic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17657) export/import for MM tables is broken
[ https://issues.apache.org/jira/browse/HIVE-17657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17657: Attachment: HIVE-17657.10.patch > export/import for MM tables is broken > - > > Key: HIVE-17657 > URL: https://issues.apache.org/jira/browse/HIVE-17657 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Attachments: HIVE-17657.01.patch, HIVE-17657.02.patch, > HIVE-17657.03.patch, HIVE-17657.04.patch, HIVE-17657.05.patch, > HIVE-17657.06.patch, HIVE-17657.07.patch, HIVE-17657.08.patch, > HIVE-17657.09.patch, HIVE-17657.10.patch, HIVE-17657.patch > > > there is mm_exim.q but it's not clear from the tests what file structure it > creates > On import the txnids in the directory names would have to be remapped if > importing to a different cluster. Perhaps export can be smart and export > highest base_x and accretive deltas (minus aborted ones). Then import can > ...? It would have to remap txn ids from the archive to new txn ids. This > would then mean that import is made up of several transactions rather than 1 > atomic op. (all locks must belong to a transaction) > One possibility is to open a new txn for each dir in the archive (where > start/end txn of file name is the same) and commit all of them at once (need > new TMgr API for that). This assumes using a shared lock (if any!) and thus > allows other inserts (not related to import) to occur. > What if you have delta_6_9, such as a result of concatenate? If we stipulate > that this must mean that there is no delta_6_6 or any other "obsolete" delta > in the archive we can map it to a new single txn delta_x_x. > Add read_only mode for tables (useful in general, may be needed for upgrade > etc) and use that to make the above atomic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19310) Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
[
https://issues.apache.org/jira/browse/HIVE-19310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466402#comment-16466402
]
Sergey Shelukhin commented on HIVE-19310:
-
Committed to master for now
> Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might
> need to be run only in test env
> -
>
> Key: HIVE-19310
> URL: https://issues.apache.org/jira/browse/HIVE-19310
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19310.02.patch, HIVE-19310.03.patch,
> HIVE-19310.1.patch
>
>
> MetaStoreDirectSql.ensureDbInit has the following 2 calls which we have
> observed taking a long time in our testing:
> {code}
> initQueries.add(pm.newQuery(MNotificationLog.class, "dbName == ''"));
> initQueries.add(pm.newQuery(MNotificationNextId.class, "nextEventId < -1"));
> {code}
> In a production environment, these tables should be initialized using
> schematool, however in a test environment, these calls might be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19410) don't create serde reader in LLAP if there's no cache
[ https://issues.apache.org/jira/browse/HIVE-19410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19410: Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Committed to master. [~mmccline] fyi this may fix some test failures introduced by vectorization by default patch (there was similar looking NPE iirc but I can't find the JIRA now); I have not committed it to branch-3. > don't create serde reader in LLAP if there's no cache > - > > Key: HIVE-19410 > URL: https://issues.apache.org/jira/browse/HIVE-19410 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19410.patch > > > Seems to crop up in some tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19310) Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
[
https://issues.apache.org/jira/browse/HIVE-19310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466395#comment-16466395
]
Vineet Garg commented on HIVE-19310:
+1 for branch-3
> Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might
> need to be run only in test env
> -
>
> Key: HIVE-19310
> URL: https://issues.apache.org/jira/browse/HIVE-19310
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19310.02.patch, HIVE-19310.03.patch,
> HIVE-19310.1.patch
>
>
> MetaStoreDirectSql.ensureDbInit has the following 2 calls which we have
> observed taking a long time in our testing:
> {code}
> initQueries.add(pm.newQuery(MNotificationLog.class, "dbName == ''"));
> initQueries.add(pm.newQuery(MNotificationNextId.class, "nextEventId < -1"));
> {code}
> In a production environment, these tables should be initialized using
> schematool, however in a test environment, these calls might be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-14388) Add number of rows inserted message after insert command in Beeline
[ https://issues.apache.org/jira/browse/HIVE-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-14388: Status: Patch Available (was: In Progress) > Add number of rows inserted message after insert command in Beeline > --- > > Key: HIVE-14388 > URL: https://issues.apache.org/jira/browse/HIVE-14388 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vihang Karajgaonkar >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Minor > Attachments: HIVE-14388-WIP.patch, HIVE-14388.02.patch, > HIVE-14388.03.patch, HIVE-14388.05.patch, HIVE-14388.06.patch, > HIVE-14388.07.patch, HIVE-14388.08.patch > > > Currently, when you run insert command on beeline, it returns a message > saying "No rows affected .." > A better and more intuitive msg would be "xxx rows inserted (26.068 seconds)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14388) Add number of rows inserted message after insert command in Beeline
[ https://issues.apache.org/jira/browse/HIVE-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-14388: Status: In Progress (was: Patch Available) > Add number of rows inserted message after insert command in Beeline > --- > > Key: HIVE-14388 > URL: https://issues.apache.org/jira/browse/HIVE-14388 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vihang Karajgaonkar >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Minor > Attachments: HIVE-14388-WIP.patch, HIVE-14388.02.patch, > HIVE-14388.03.patch, HIVE-14388.05.patch, HIVE-14388.06.patch, > HIVE-14388.07.patch, HIVE-14388.08.patch > > > Currently, when you run insert command on beeline, it returns a message > saying "No rows affected .." > A better and more intuitive msg would be "xxx rows inserted (26.068 seconds)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18748) Rename table impacts the ACID behaviour as table names are not updated in meta-tables.
[ https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18748: -- Fix Version/s: (was: 3.1.0) 3.0.0 > Rename table impacts the ACID behaviour as table names are not updated in > meta-tables. > -- > > Key: HIVE-18748 > URL: https://issues.apache.org/jira/browse/HIVE-18748 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Eugene Koifman >Priority: Critical > Labels: ACID, DDL > Fix For: 3.0.0 > > > ACID implementation uses metatables such as TXN_COMPONENTS, > COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to > manage ACID operations. > Per table write ID implementation (HIVE-18192) introduces couple of > metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids > allocated per table. > Now, when we rename any tables, it is necessary to update the corresponding > table names in these metatables as well. Otherwise, ACID table operations > won't work properly. > Since, this change is significant and have other side-effects, we propose to > disable rename tables on ACID tables until a fix is figured out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
