[jira] [Commented] (HIVE-14564) Column Pruning generates out of order columns in SelectOperator which cause ArrayIndexOutOfBoundsException.
[
https://issues.apache.org/jira/browse/HIVE-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136371#comment-17136371
]
Jl.Yang commented on HIVE-14564:
Hi，In version 0.98, I select result by left join , in the first sub query I
used order by key word to sort query result. It happend the same error, when I
delete order by key word , the error is disapper .I want to know if it is the
same question by serialization LazyBinarySerDe.
2020-06-16 11:01:21,128 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report
from attempt_1587475402360_2625484_m_000384_3: Error:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{"_col0":29,"_col1":"_vivoX6PlusDEES8F6IFIFLBPFVK\u001dN_vivoX6PlusDEES8F6IFIFLBPFVK�]:�ڈ\u0011r
\u�F�{\nu","_col2":"�K�\u0006&\u0006�\u00014<�70AE�\u0006��\u0011�#��Dp\u�&��\u0011���\u0006�\u00014<�\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u","_col3":70,"_col4":76,"_col5":66,"_col6":null,"_col7":null,"_col8":null,"_col9":null,"_col10":null,"_col11":null,"_col12":70,"_col13":null,"_col14":86,"_col15":null,"_col16":156413}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
{"_col0":29,"_col1":"_vivoX6PlusDEES8F6IFIFLBPFVK\u001dN_vivoX6PlusDEES8F6IFIFLBPFVK�]:�ڈ\u0011r
\u�F�{\nu","_col2":"�K�\u0006&\u0006�\u00014<�70AE�\u0006��\u0011�#��Dp\u�&��\u0011���\u0006�\u00014<�\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u","_col3":70,"_col4":76,"_col5":66,"_col6":null,"_col7":null,"_col8":null,"_col9":null,"_col10":null,"_col11":null,"_col12":70,"_col13":null,"_col14":86,"_col15":null,"_col16":156413}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ArrayIndexOutOfBoundsException
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:329)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:539)
... 9 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.io.Text.set(Text.java:225)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:261)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:199)
at
org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.populateCachedDistributionKeys(ReduceSinkOperator.java:349)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:282)
... 13 more
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> ---
>
> Key: HIVE-14564
> URL: https://issues.apache.org/jira/browse/HIVE-14564
> Project: Hive
> Issue Type: Bug
> Components: Query
[jira] [Work logged] (HIVE-22957) Add Option For Predicate Filtering In MSCK REPAIR TABLE Command
[ https://issues.apache.org/jira/browse/HIVE-22957?focusedWorklogId=446284&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446284 ] ASF GitHub Bot logged work on HIVE-22957: - Author: ASF GitHub Bot Created on: 16/Jun/20 04:48 Start Date: 16/Jun/20 04:48 Worklog Time Spent: 10m Work Description: shameersss1 commented on pull request #1105: URL: https://github.com/apache/hive/pull/1105#issuecomment-644528808 @kgyrtkirk @jcamachor @prasanthj Could you please review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446284) Time Spent: 1h 10m (was: 1h) > Add Option For Predicate Filtering In MSCK REPAIR TABLE Command > --- > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: Design Doc_ Partition Filtering In MSCK REPAIR > TABLE.pdf, HIVE-22957.01.patch, HIVE-22957.02.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > *Design Doc:* > [^Design Doc_ Partition Filtering In MSCK REPAIR TABLE.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23612) Option for HiveStrictManagedMigration to impersonate a user for FS operations
[ https://issues.apache.org/jira/browse/HIVE-23612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136278#comment-17136278 ] Jason Dere commented on HIVE-23612: --- I think this looks good. One suggestion - rather than hardcoding "hive" as the default owner in these changes, there is a setting "strict.managed.tables.migration.owner" that is used for the managed table owner .. you could use a similar setting "strict.external.tables.migration.owner" for this "unowned" tables user, with a default value of "hive". > Option for HiveStrictManagedMigration to impersonate a user for FS operations > - > > Key: HIVE-23612 > URL: https://issues.apache.org/jira/browse/HIVE-23612 > Project: Hive > Issue Type: Improvement >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-23612.0.patch, HIVE-23612.1.patch > > > HiveStrictManagedMigration tool can be used to move HDFS paths and to change > ownership on said paths. It may be beneficial to do such file system > operations as a different user than the one the tool itself is run. > Moreover, while creating the external DB directory, the tool will chown the > new directory to the user set as DB owner in HMS. If this is unset, no chown > command is used. In this case we should make the 'hive' user the directory > owner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23627) Review of GroupByOperator
[ https://issues.apache.org/jira/browse/HIVE-23627?focusedWorklogId=446225&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446225 ] ASF GitHub Bot logged work on HIVE-23627: - Author: ASF GitHub Bot Created on: 16/Jun/20 01:35 Start Date: 16/Jun/20 01:35 Worklog Time Spent: 10m Work Description: belugabehr opened a new pull request #1067: URL: https://github.com/apache/hive/pull/1067 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446225) Time Spent: 2h 20m (was: 2h 10m) > Review of GroupByOperator > - > > Key: HIVE-23627 > URL: https://issues.apache.org/jira/browse/HIVE-23627 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23627) Review of GroupByOperator
[ https://issues.apache.org/jira/browse/HIVE-23627?focusedWorklogId=446224&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446224 ] ASF GitHub Bot logged work on HIVE-23627: - Author: ASF GitHub Bot Created on: 16/Jun/20 01:34 Start Date: 16/Jun/20 01:34 Worklog Time Spent: 10m Work Description: belugabehr closed pull request #1067: URL: https://github.com/apache/hive/pull/1067 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446224) Time Spent: 2h 10m (was: 2h) > Review of GroupByOperator > - > > Key: HIVE-23627 > URL: https://issues.apache.org/jira/browse/HIVE-23627 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23699) Cleanup HIVEQUERYRESULTFILEFORMAT handling
[
https://issues.apache.org/jira/browse/HIVE-23699?focusedWorklogId=446208&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446208
]
ASF GitHub Bot logged work on HIVE-23699:
-
Author: ASF GitHub Bot
Created on: 16/Jun/20 00:58
Start Date: 16/Jun/20 00:58
Worklog Time Spent: 10m
Work Description: jfsii commented on a change in pull request #1119:
URL: https://github.com/apache/hive/pull/1119#discussion_r440522788
##
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##
@@ -101,6 +102,65 @@
private volatile boolean isSparkConfigUpdated = false;
private static final int LOG_PREFIX_LENGTH = 64;
+ interface HiveConfEnum> {
+public static > T from(Class cls, String value, T
invalidEnum) {
+ try {
+return T.valueOf(cls, value.toUpperCase());
+ } catch (Exception e) {
+return invalidEnum;
+ }
+}
+ }
Review comment:
Should I even bother with this interface? The intent was originally to
have static definitions of getValidSet and getInvalid as part of the interface
in addition to from. Except Java interfaces don't allow implementations of an
interface to override static methods. The from method here is also not super
useful in that I think I still have to fully qualify the call (I.E.
HiveConfEnum.from()). It would be nice if we could define required static
methods on a enum interface for future uses of enums in HiveConf, but I don't
think it is cleanly possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446208)
Time Spent: 20m (was: 10m)
> Cleanup HIVEQUERYRESULTFILEFORMAT handling
> --
>
> Key: HIVE-23699
> URL: https://issues.apache.org/jira/browse/HIVE-23699
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> HIVEQUERYRESULTFILEFORMAT format handling has grown over the years and has
> become somewhat messy code wise in SemanticAnalyzer and TaskCompiler. There
> are special cases where the HIVEQUERYRESULTFILEFORMAT setting gets changed at
> runtime that may cause issues if a user changes execution.engines between
> queries and probably other corner cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21880) Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
[
https://issues.apache.org/jira/browse/HIVE-21880?focusedWorklogId=446188&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446188
]
ASF GitHub Bot logged work on HIVE-21880:
-
Author: ASF GitHub Bot
Created on: 16/Jun/20 00:24
Start Date: 16/Jun/20 00:24
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #684:
URL: https://github.com/apache/hive/pull/684
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446188)
Time Spent: 3h 20m (was: 3h 10m)
> Enable flaky test
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
> ---
>
> Key: HIVE-21880
> URL: https://issues.apache.org/jira/browse/HIVE-21880
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21880.01.patch, HIVE-21880.02.patch,
> HIVE-21880.03.patch, HIVE-21880.04.patch, HIVE-21880.05.patch,
> HIVE-21880.06.patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Need tp enable
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
> which is disabled as it is flaky and randomly failing with below error.
> {code}
> Error Message
> Notification events are missing in the meta store.
> Stacktrace
> java.lang.IllegalStateException: Notification events are missing in the meta
> store.
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:3246)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy58.getNextNotification(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:107)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:159)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.hasNext(EventUtils.java:189)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:231)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.execute(ReplDumpTask.java:121)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.run(WarehouseInstance.java:227)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:282)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:265)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:289)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites(TestReplicationScenariosAcidTablesBootstrap.java:328)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
>
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[ https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=446191&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446191 ] ASF GitHub Bot logged work on HIVE-21764: - Author: ASF GitHub Bot Created on: 16/Jun/20 00:24 Start Date: 16/Jun/20 00:24 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #679: URL: https://github.com/apache/hive/pull/679 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446191) Time Spent: 7.5h (was: 7h 20m) > REPL DUMP should detect and bootstrap any rename table events where old table > was excluded but renamed table is included. > - > > Key: HIVE-21764 > URL: https://issues.apache.org/jira/browse/HIVE-21764 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sankar Hariappan >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch, > HIVE-21764.03.patch > > Time Spent: 7.5h > Remaining Estimate: 0h > > REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular > expression + inclusion/exclusion list. So, in case of rename table event, the > event will be ignored if old table doesn't match the pattern but the new > table should be bootstrapped. REPL DUMP should have a mechanism to detect > such tables and automatically bootstrap with incremental replication.Also, if > renamed table is excluded from replication policy, then need to drop the old > table at target as well. > There are 4 scenarios that needs to be handled. > # Both new name and old name satisfies the table name pattern filter. > ## No need to do anything. The incremental event for rename should take care > of the replication. > # Both the names does not satisfy the table name pattern filter. > ## Both the names are not in the scope of the policy and thus nothing needs > to be done. > # New name satisfies the pattern but the old name does not. > ## The table will not be present at the target. > ## Rename event handler for dump should detect this case and add the new > table name to the list of table for bootstrap. > ## All the events related to the table (new name) should be ignored. > ## If there is a drop event for the table (with new name), then remove the > table from the list of tables to be bootstrapped. > ## In case of rename (double rename) > ### If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ### If the new name does not satisfies then just removed the table name from > the list of tables to be bootstrapped. > # New name does not satisfies the pattern but the old name satisfies. > ## Change the rename event to a drop event. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21829) HiveMetaStore authorization issue with AlterTable and DropTable events
[ https://issues.apache.org/jira/browse/HIVE-21829?focusedWorklogId=446195&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446195 ] ASF GitHub Bot logged work on HIVE-21829: - Author: ASF GitHub Bot Created on: 16/Jun/20 00:24 Start Date: 16/Jun/20 00:24 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #658: URL: https://github.com/apache/hive/pull/658 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446195) Time Spent: 0.5h (was: 20m) > HiveMetaStore authorization issue with AlterTable and DropTable events > -- > > Key: HIVE-21829 > URL: https://issues.apache.org/jira/browse/HIVE-21829 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21829.1.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > With HIVE-21753, we have HiveMetastore authorizer which uses HiveAuthorizer > interface to authorizer metastore events. > This jira is to fix a bug in HIVE-21753 which failed to authorizer Alter and > DropTable events -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21877) Change HCatTableInfo to not be transient in PartInfo
[ https://issues.apache.org/jira/browse/HIVE-21877?focusedWorklogId=446194&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446194 ] ASF GitHub Bot logged work on HIVE-21877: - Author: ASF GitHub Bot Created on: 16/Jun/20 00:24 Start Date: 16/Jun/20 00:24 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #672: URL: https://github.com/apache/hive/pull/672 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446194) Time Spent: 40m (was: 0.5h) > Change HCatTableInfo to not be transient in PartInfo > > > Key: HIVE-21877 > URL: https://issues.apache.org/jira/browse/HIVE-21877 > Project: Hive > Issue Type: New Feature >Reporter: Ankit Jhalaria >Assignee: Ankit Jhalaria >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Since HCatTableInfo is serializable, removing the transient annotation from > it. We were running into NPE during serialization while using HCatalogIO with > Beam. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21841) Leader election in HMS to run housekeeping tasks.
[ https://issues.apache.org/jira/browse/HIVE-21841?focusedWorklogId=446189&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446189 ] ASF GitHub Bot logged work on HIVE-21841: - Author: ASF GitHub Bot Created on: 16/Jun/20 00:24 Start Date: 16/Jun/20 00:24 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #668: URL: https://github.com/apache/hive/pull/668 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446189) Time Spent: 3h 50m (was: 3h 40m) > Leader election in HMS to run housekeeping tasks. > - > > Key: HIVE-21841 > URL: https://issues.apache.org/jira/browse/HIVE-21841 > Project: Hive > Issue Type: New Feature >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21841.01.patch, HIVE-21841.02.patch, > HIVE-21841.04.patch, HIVE-21841.05.patch, HIVE-21841.06.patch, > HIVE-21841.07.patch, HIVE-21841.08.patch, HIVE-21841.09.patch > > Time Spent: 3h 50m > Remaining Estimate: 0h > > HMS performs housekeeping tasks. When there are multiple HMSes we need to > have a leader HMS elected which will carry out those housekeeping tasks. > These tasks include execution of compaction tasks, auto-discovering > partitions for external tables, generation of compaction tasks, repl thread > etc. > Note that, though the code for compaction tasks, auto-discovery of partitions > etc. is in Hive, the actual tasks are initiated by an HMS configured to do > so. So, leader election is required only for HMS and not for HS2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21788) Support replication from haddop-2 (hive 3.0 and beelow) on-prem cluster to hadoop-3 (hive 4 and above) cloud cluster
[ https://issues.apache.org/jira/browse/HIVE-21788?focusedWorklogId=446193&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446193 ] ASF GitHub Bot logged work on HIVE-21788: - Author: ASF GitHub Bot Created on: 16/Jun/20 00:24 Start Date: 16/Jun/20 00:24 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #666: URL: https://github.com/apache/hive/pull/666 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446193) Time Spent: 0.5h (was: 20m) > Support replication from haddop-2 (hive 3.0 and beelow) on-prem cluster to > hadoop-3 (hive 4 and above) cloud cluster > > > Key: HIVE-21788 > URL: https://issues.apache.org/jira/browse/HIVE-21788 > Project: Hive > Issue Type: Task > Components: HiveServer2, repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21788.01.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > In case of replication to cloud both dump and load are executed in the source > cluster. This push based replication is done to avoid computation at target > cloud cluster. In case in the source cluster, strict managed table is not set > to true the tables will be non acid. So during replication to a cluster with > strict managed table, migration logic same as upgrade tool has to be applied > on the replicated data. This migration logic is implemented only in hive4.0. > So it's required that a hive 4.0 instance started at the source cluster. If > the source cluster has hadoop-2 installation, hive4 has to be built with > hadoop-2 and necessary changes are required in the pom files and the shim > files. > 1. Change the pom.xml files to accept a profile for hadoop-2. If hadoop-2 > profile is set, the hadoop version should be set accordingly to hadoop-2. > 2. In shim creare a new file for hadoop-2. Based on the profile the > respective file will be included in the build. > 3. Changed artifactId hadoop-hdfs-client to hadoop-client as in hadoop-2 the > jars are stored under hadoop-client folder. > > > Command to enable hadop-2 dependency — mvn clean install package > -DskipTests -Pdist -pl '!standalone-metastore, !llap-common, !llap-client, > !llap-ext-client, !llap-tez, !llap-server, !hbase-handler, !service, !hplsql, > !kryo-registrator' -Phadoop-2.7 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21844) HMS schema Upgrade Script is failing with NPE
[ https://issues.apache.org/jira/browse/HIVE-21844?focusedWorklogId=446190&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446190 ] ASF GitHub Bot logged work on HIVE-21844: - Author: ASF GitHub Bot Created on: 16/Jun/20 00:24 Start Date: 16/Jun/20 00:24 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #667: URL: https://github.com/apache/hive/pull/667 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446190) Time Spent: 1h (was: 50m) > HMS schema Upgrade Script is failing with NPE > - > > Key: HIVE-21844 > URL: https://issues.apache.org/jira/browse/HIVE-21844 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21844.01.patch, HIVE-21844.02.patch > > Time Spent: 1h > Remaining Estimate: 0h > > schema upgrade tool is failing with NPE while executing "SELECT 'Upgrading > MetaStore schema from 1.2.0 to 2.0.0' AS ' '". The header row (metadata) is > coming with rows having value null. This is causing null pointer access in > function TableOutputFormat::getOutputString when row.values[i] is accessed. > Instead of " AS ' ' ", if some other value like "AS dummy" is given, it's > working fine. The issue is coming with mysql version 5.7 and above. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21864) LlapBaseInputFormat#closeAll() throws ConcurrentModificationException
[ https://issues.apache.org/jira/browse/HIVE-21864?focusedWorklogId=446192&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446192 ] ASF GitHub Bot logged work on HIVE-21864: - Author: ASF GitHub Bot Created on: 16/Jun/20 00:24 Start Date: 16/Jun/20 00:24 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #669: URL: https://github.com/apache/hive/pull/669 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446192) Time Spent: 2h (was: 1h 50m) > LlapBaseInputFormat#closeAll() throws ConcurrentModificationException > - > > Key: HIVE-21864 > URL: https://issues.apache.org/jira/browse/HIVE-21864 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21864.1.patch, HIVE-21864.2.patch, > HIVE-21864.3.patch, HIVE-21864.4.patch, HIVE-21864.5.patch > > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21737) Upgrade Avro to version 1.9.2
[ https://issues.apache.org/jira/browse/HIVE-21737?focusedWorklogId=446196&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446196 ] ASF GitHub Bot logged work on HIVE-21737: - Author: ASF GitHub Bot Created on: 16/Jun/20 00:24 Start Date: 16/Jun/20 00:24 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #674: URL: https://github.com/apache/hive/pull/674 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446196) Time Spent: 1h 20m (was: 1h 10m) > Upgrade Avro to version 1.9.2 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.2.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Avro 1.9.2 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=446187&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446187 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 16/Jun/20 00:24 Start Date: 16/Jun/20 00:24 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #688: URL: https://github.com/apache/hive/pull/688 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446187) Time Spent: 11.5h (was: 11h 20m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch, HIVE-21886.04.patch, HIVE-21886.04.patch, > HIVE-21886.05.patch > > Time Spent: 11.5h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. If the old table does not match the old policy drop it, even if the > table is not present at target. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > 2. If the new policy does not match the table name, then ignore the > event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be ignored. But rename handler should decide based on new name.The old table > name will not be returned by get-all-table, so replace handler will not d > anything for the old table. > ### New name not matching old policy > New name not matching new policy > * As
[jira] [Updated] (HIVE-23699) Cleanup HIVEQUERYRESULTFILEFORMAT handling
[ https://issues.apache.org/jira/browse/HIVE-23699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23699: -- Labels: pull-request-available (was: ) > Cleanup HIVEQUERYRESULTFILEFORMAT handling > -- > > Key: HIVE-23699 > URL: https://issues.apache.org/jira/browse/HIVE-23699 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HIVEQUERYRESULTFILEFORMAT format handling has grown over the years and has > become somewhat messy code wise in SemanticAnalyzer and TaskCompiler. There > are special cases where the HIVEQUERYRESULTFILEFORMAT setting gets changed at > runtime that may cause issues if a user changes execution.engines between > queries and probably other corner cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23699) Cleanup HIVEQUERYRESULTFILEFORMAT handling
[ https://issues.apache.org/jira/browse/HIVE-23699?focusedWorklogId=446186&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446186 ] ASF GitHub Bot logged work on HIVE-23699: - Author: ASF GitHub Bot Created on: 16/Jun/20 00:20 Start Date: 16/Jun/20 00:20 Worklog Time Spent: 10m Work Description: jfsii opened a new pull request #1119: URL: https://github.com/apache/hive/pull/1119 Change-Id: I8ce7061fde58fc94ee4750142f699e3cc23980fd ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HIVE-X: Fix a typo in YYY) For more details, please see https://cwiki.apache.org/confluence/display/Hive/HowToContribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446186) Remaining Estimate: 0h Time Spent: 10m > Cleanup HIVEQUERYRESULTFILEFORMAT handling > -- > > Key: HIVE-23699 > URL: https://issues.apache.org/jira/browse/HIVE-23699 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > HIVEQUERYRESULTFILEFORMAT format handling has grown over the years and has > become somewhat messy code wise in SemanticAnalyzer and TaskCompiler. There > are special cases where the HIVEQUERYRESULTFILEFORMAT setting gets changed at > runtime that may cause issues if a user changes execution.engines between > queries and probably other corner cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23683) Add queue time to compaction
[
https://issues.apache.org/jira/browse/HIVE-23683?focusedWorklogId=446144&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446144
]
ASF GitHub Bot logged work on HIVE-23683:
-
Author: ASF GitHub Bot
Created on: 15/Jun/20 21:51
Start Date: 15/Jun/20 21:51
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #1101:
URL: https://github.com/apache/hive/pull/1101#discussion_r440467302
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java
##
@@ -81,6 +81,8 @@ private void writeHeader(DataOutputStream os) throws
IOException {
os.write(Utilities.tabCode);
os.writeBytes("Worker");
os.write(Utilities.tabCode);
+os.writeBytes("Queue Time");
Review comment:
How about enqueue time? Short, easy to understand...
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
##
@@ -116,6 +117,7 @@ public String toString() {
"partName:" + partName + "," +
"state:" + state + "," +
"type:" + type + "," +
+ "queueTime:" + queueTime + "," +
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446144)
Time Spent: 40m (was: 0.5h)
> Add queue time to compaction
>
>
> Key: HIVE-23683
> URL: https://issues.apache.org/jira/browse/HIVE-23683
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> It would be good to report to the user when the transaction is initiated.
> This info can be used when considering the health status of the compaction
> system
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23683) Add queue time to compaction
[ https://issues.apache.org/jira/browse/HIVE-23683?focusedWorklogId=446142&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446142 ] ASF GitHub Bot logged work on HIVE-23683: - Author: ASF GitHub Bot Created on: 15/Jun/20 21:48 Start Date: 15/Jun/20 21:48 Worklog Time Spent: 10m Work Description: pvary commented on a change in pull request #1101: URL: https://github.com/apache/hive/pull/1101#discussion_r440465947 ## File path: ql/src/test/queries/clientpositive/sysdb.q ## @@ -1,5 +1,6 @@ --! qt:dataset:src,part,srcbucket:ONLY --! qt:sysdb +--! qt:replace:/(initiated\s+NULL\s+NULL\s+)[0-9]*(\s+NULL)/$1#Masked#$2/ Review comment: Done ## File path: ql/src/test/queries/clientpositive/dbtxnmgr_showlocks.q ## @@ -1,3 +1,4 @@ + --! qt:replace:/(initiated\s+---\s+---\s+)[0-9]*(\s+---)/$1#Masked#$2/ Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446142) Time Spent: 0.5h (was: 20m) > Add queue time to compaction > > > Key: HIVE-23683 > URL: https://issues.apache.org/jira/browse/HIVE-23683 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > It would be good to report to the user when the transaction is initiated. > This info can be used when considering the health status of the compaction > system -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23592) Routine "makeIntPair" is Not Correct
[
https://issues.apache.org/jira/browse/HIVE-23592?focusedWorklogId=446126&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446126
]
ASF GitHub Bot logged work on HIVE-23592:
-
Author: ASF GitHub Bot
Created on: 15/Jun/20 21:14
Start Date: 15/Jun/20 21:14
Worklog Time Spent: 10m
Work Description: miklosgergely commented on a change in pull request
#1052:
URL: https://github.com/apache/hive/pull/1052#discussion_r440255453
##
File path: common/src/test/org/apache/hadoop/hive/common/TestNumberUtils.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * Test suite for the {@link NumberUtils} class.
+ */
+public class TestNumberUtils {
+
+ @Test
+ public void testMinLong() {
+final long pair = NumberUtils.makeIntPair(Integer.MIN_VALUE,
Integer.MIN_VALUE);
+Assert.assertEquals(Integer.MIN_VALUE, NumberUtils.getFirstInt(pair));
+Assert.assertEquals(Integer.MIN_VALUE, NumberUtils.getSecondInt(pair));
+ }
+
+ @Test
+ public void testMaxLong() {
+final long pair = NumberUtils.makeIntPair(Integer.MAX_VALUE,
Integer.MAX_VALUE);
+Assert.assertEquals(Integer.MAX_VALUE, NumberUtils.getFirstInt(pair));
+Assert.assertEquals(Integer.MAX_VALUE, NumberUtils.getSecondInt(pair));
+ }
+
+ @Test
+ public void testZeroLong() {
+final long pair = NumberUtils.makeIntPair(0, 0);
+Assert.assertEquals(0, NumberUtils.getFirstInt(pair));
+Assert.assertEquals(0, NumberUtils.getSecondInt(pair));
+ }
+
Review comment:
Please add some more tests with "everyday" positive and negative
numbers, the current tests prove only that it works fine fo the extreme values.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446126)
Time Spent: 1.5h (was: 1h 20m)
> Routine "makeIntPair" is Not Correct
>
>
> Key: HIVE-23592
> URL: https://issues.apache.org/jira/browse/HIVE-23592
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> {code:java|title=BuddyAllocator.java}
> // Utility methods used to store pairs of ints as long.
> private static long makeIntPair(int first, int second) {
> return ((long)first) << 32 | second;
> }
> private static int getFirstInt(long result) {
> return (int) (result >>> 32);
> }
> private static int getSecondInt(long result) {
> return (int) (result & ((1L << 32) - 1));
> }
> {code}
> {code:java}
> long result = LLAP.makeIntPair(Integer.MIN_VALUE, Integer.MIN_VALUE);
> if (LLAP.getFirstInt(result) != Integer.MIN_VALUE) {
>throw new Exception();
> }
> if (LLAP.getSecondInt(result) != Integer.MIN_VALUE) {
> throw new Exception();
>}
> /*
> * Exception in thread "main" java.lang.Exception
> *at org.test.TestMe.main(TestMe.java:19)
> */
> {code}
> [https://github.com/apache/hive/blob/4b670877c280b37c5776046f66d766079489b2a8/llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java#L1677]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21952) Hive should allow to delete serde properties too, not just add them
[ https://issues.apache.org/jira/browse/HIVE-21952?focusedWorklogId=446109&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446109 ] ASF GitHub Bot logged work on HIVE-21952: - Author: ASF GitHub Bot Created on: 15/Jun/20 20:47 Start Date: 15/Jun/20 20:47 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #1112: URL: https://github.com/apache/hive/pull/1112#issuecomment-644380907 Requested tests are added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446109) Time Spent: 20m (was: 10m) > Hive should allow to delete serde properties too, not just add them > --- > > Key: HIVE-21952 > URL: https://issues.apache.org/jira/browse/HIVE-21952 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0, 4.0.0, 2.3.5 >Reporter: Ruslan Dautkhanov >Assignee: Miklos Gergely >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Hive should allow to delete serde properties not just add/change them > We have a use case when a presence of certain serde properties > causes issues and we want to delete just that one serde property. > It's not currently possible. > Thanks. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?focusedWorklogId=446101&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446101 ] ASF GitHub Bot logged work on HIVE-23363: - Author: ASF GitHub Bot Created on: 15/Jun/20 20:36 Start Date: 15/Jun/20 20:36 Worklog Time Spent: 10m Work Description: belugabehr opened a new pull request #1118: URL: https://github.com/apache/hive/pull/1118 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446101) Time Spent: 0.5h (was: 20m) > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23363.2.patch, HIVE-23363.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been > retired: > [http://www.datanucleus.org/documentation/products.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?focusedWorklogId=446099&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446099 ] ASF GitHub Bot logged work on HIVE-23363: - Author: ASF GitHub Bot Created on: 15/Jun/20 20:35 Start Date: 15/Jun/20 20:35 Worklog Time Spent: 10m Work Description: belugabehr closed pull request #1118: URL: https://github.com/apache/hive/pull/1118 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446099) Time Spent: 20m (was: 10m) > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23363.2.patch, HIVE-23363.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been > retired: > [http://www.datanucleus.org/documentation/products.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23363: -- Labels: pull-request-available (was: ) > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23363.2.patch, HIVE-23363.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been > retired: > [http://www.datanucleus.org/documentation/products.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?focusedWorklogId=446026&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446026 ] ASF GitHub Bot logged work on HIVE-23363: - Author: ASF GitHub Bot Created on: 15/Jun/20 18:58 Start Date: 15/Jun/20 18:58 Worklog Time Spent: 10m Work Description: belugabehr opened a new pull request #1118: URL: https://github.com/apache/hive/pull/1118 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446026) Remaining Estimate: 0h Time Spent: 10m > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Critical > Attachments: HIVE-23363.2.patch, HIVE-23363.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been > retired: > [http://www.datanucleus.org/documentation/products.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-19624) the method closeSession of SessionManager has a synchronized, is it able to be removed?
[ https://issues.apache.org/jira/browse/HIVE-19624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam resolved HIVE-19624. -- Fix Version/s: 4.0.0 Resolution: Duplicate Looks like this issue has been addressed via HIVE-22165. Closing this jira as duplicate. > the method closeSession of SessionManager has a synchronized, is it able to > be removed? > > > Key: HIVE-19624 > URL: https://issues.apache.org/jira/browse/HIVE-19624 > Project: Hive > Issue Type: Wish > Components: HiveServer2 >Affects Versions: 2.1.1 >Reporter: xulongfetion >Assignee: Naveen Gangam >Priority: Minor > Fix For: 4.0.0 > > > The methods closeSession of > org.apache.hive.service.cli.session.SessionManager use > synchronized for thread safe, i looked at the code and wonder why there's > need for synchronized. > Because closeSession sometimes use the hadoop ipc call and takes too much > time to response, so if it is possible to remove the keyword synchronized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23388) CTAS queries should use target's location for staging.
[ https://issues.apache.org/jira/browse/HIVE-23388?focusedWorklogId=445978&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445978 ] ASF GitHub Bot logged work on HIVE-23388: - Author: ASF GitHub Bot Created on: 15/Jun/20 18:03 Start Date: 15/Jun/20 18:03 Worklog Time Spent: 10m Work Description: nrg4878 opened a new pull request #1117: URL: https://github.com/apache/hive/pull/1117 …veen Gangam) ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HIVE-X: Fix a typo in YYY) For more details, please see https://cwiki.apache.org/confluence/display/Hive/HowToContribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445978) Remaining Estimate: 0h Time Spent: 10m > CTAS queries should use target's location for staging. > -- > > Key: HIVE-23388 > URL: https://issues.apache.org/jira/browse/HIVE-23388 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In cloud based storage systems, renaming files across different root level > buckets seem to be disallowed. The S3AFileSystem throws the following > exception. This appears to be bug in S3FS impl. > Failed with exception Wrong FS > s3a://hive-managed/clusters/env-x/warehouse--/warehouse/tablespace/managed/hive/tpch.db/customer/delta_001_001_ > -expected s3a://hive-external > 2020-04-27T19:34:27,573 INFO [Thread-6] jdbc.TestDriver: > java.lang.IllegalArgumentException: Wrong FS > s3a://hive-managed//clusters/env-/warehouse--/warehouse/tablespace/managed/hive/tpch.db/customer/delta_001_001_ > -expected s3a://hive-external > But we should fix our query plans to use the target table's directory for > staging as well. That should resolve this issue and it is the right thing to > do as well (in case there are different encryption zones/keys for these > buckets). > Fix in HIVE-22995 probably changed this behavior. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23388) CTAS queries should use target's location for staging.
[ https://issues.apache.org/jira/browse/HIVE-23388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23388: -- Labels: pull-request-available (was: ) > CTAS queries should use target's location for staging. > -- > > Key: HIVE-23388 > URL: https://issues.apache.org/jira/browse/HIVE-23388 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In cloud based storage systems, renaming files across different root level > buckets seem to be disallowed. The S3AFileSystem throws the following > exception. This appears to be bug in S3FS impl. > Failed with exception Wrong FS > s3a://hive-managed/clusters/env-x/warehouse--/warehouse/tablespace/managed/hive/tpch.db/customer/delta_001_001_ > -expected s3a://hive-external > 2020-04-27T19:34:27,573 INFO [Thread-6] jdbc.TestDriver: > java.lang.IllegalArgumentException: Wrong FS > s3a://hive-managed//clusters/env-/warehouse--/warehouse/tablespace/managed/hive/tpch.db/customer/delta_001_001_ > -expected s3a://hive-external > But we should fix our query plans to use the target table's directory for > staging as well. That should resolve this issue and it is the right thing to > do as well (in case there are different encryption zones/keys for these > buckets). > Fix in HIVE-22995 probably changed this behavior. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23700) HiveConf static initialization fails when JAR URI is opaque
[ https://issues.apache.org/jira/browse/HIVE-23700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francisco Guerrero updated HIVE-23700: -- Attachment: HIVE-23700.1.patch Affects Version/s: (was: 3.1.2) Status: Patch Available (was: Open) Attached a patch to fix the issue. The patch handles IllegalArgumentExceptions thrown by the File constructor when the jar URI is not supported. This fixes the static initialization of the HiveConf class when four conditions are met: 1. hive-site.xml is not present on the classpath 2. hive-site.xml is not present on the "HIVE_CONF_DIR" directory 3. hive-site.xml is not present on the "HIVE_HOME" directory 4. jar URI is not absolute, or is opaque, or URI scheme is null or not file, uri authority is not null, uri fragment is not null, uri query is not null and finally uri path is empty. > HiveConf static initialization fails when JAR URI is opaque > --- > > Key: HIVE-23700 > URL: https://issues.apache.org/jira/browse/HIVE-23700 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.3.7 >Reporter: Francisco Guerrero >Assignee: Francisco Guerrero >Priority: Minor > Attachments: HIVE-23700.1.patch > > Original Estimate: 120h > Remaining Estimate: 120h > > HiveConf static initialization fails when the jar URI is opaque, for example > when it's embedded as a fat jar in a spring boot application. Then > initialization of the HiveConf static block fails and the HiveConf class does > not get classloaded. The opaque URI in my case looks like this > _jar:file:/usr/local/server/some-service-jar.jar!/BOOT-INF/lib/hive-common-2.3.7.jar!/_ > HiveConf#findConfigFile should be able to handle `IllegalArgumentException` > when the jar `URI` provided to `File` throws the exception. > To surface this issue three conditions need to be met. > 1. hive-site.xml should not be on the classpath > 2. hive-site.xml should not be on "HIVE_CONF_DIR" > 3. hive-site.xml should not be on "HIVE_HOME" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23700) HiveConf static initialization fails when JAR URI is opaque
[ https://issues.apache.org/jira/browse/HIVE-23700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francisco Guerrero updated HIVE-23700: -- Attachment: (was: HIVE-23700.1.patch) > HiveConf static initialization fails when JAR URI is opaque > --- > > Key: HIVE-23700 > URL: https://issues.apache.org/jira/browse/HIVE-23700 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.3.7 >Reporter: Francisco Guerrero >Assignee: Francisco Guerrero >Priority: Minor > Attachments: HIVE-23700.1.patch > > Original Estimate: 120h > Remaining Estimate: 120h > > HiveConf static initialization fails when the jar URI is opaque, for example > when it's embedded as a fat jar in a spring boot application. Then > initialization of the HiveConf static block fails and the HiveConf class does > not get classloaded. The opaque URI in my case looks like this > _jar:file:/usr/local/server/some-service-jar.jar!/BOOT-INF/lib/hive-common-2.3.7.jar!/_ > HiveConf#findConfigFile should be able to handle `IllegalArgumentException` > when the jar `URI` provided to `File` throws the exception. > To surface this issue three conditions need to be met. > 1. hive-site.xml should not be on the classpath > 2. hive-site.xml should not be on "HIVE_CONF_DIR" > 3. hive-site.xml should not be on "HIVE_HOME" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23683) Add queue time to compaction
[
https://issues.apache.org/jira/browse/HIVE-23683?focusedWorklogId=445974&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445974
]
ASF GitHub Bot logged work on HIVE-23683:
-
Author: ASF GitHub Bot
Created on: 15/Jun/20 17:55
Start Date: 15/Jun/20 17:55
Worklog Time Spent: 10m
Work Description: klcopp commented on a change in pull request #1101:
URL: https://github.com/apache/hive/pull/1101#discussion_r440343846
##
File path: ql/src/test/queries/clientpositive/dbtxnmgr_showlocks.q
##
@@ -1,3 +1,4 @@
+ --! qt:replace:/(initiated\s+---\s+---\s+)[0-9]*(\s+---)/$1#Masked#$2/
Review comment:
Nit: maybe add a comment here about what's going on, to save someone
time down the road
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java
##
@@ -81,6 +81,8 @@ private void writeHeader(DataOutputStream os) throws
IOException {
os.write(Utilities.tabCode);
os.writeBytes("Worker");
os.write(Utilities.tabCode);
+os.writeBytes("Queue Time");
Review comment:
"Queue time" sounds like "time spent in queue." Something like "Entered
queue" sounds more like a timestamp to me.
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
##
@@ -116,6 +117,7 @@ public String toString() {
"partName:" + partName + "," +
"state:" + state + "," +
"type:" + type + "," +
+ "queueTime:" + queueTime + "," +
Review comment:
(Opportunity: "start" is missing from this method)
##
File path: ql/src/test/queries/clientpositive/sysdb.q
##
@@ -1,5 +1,6 @@
--! qt:dataset:src,part,srcbucket:ONLY
--! qt:sysdb
+--! qt:replace:/(initiated\s+NULL\s+NULL\s+)[0-9]*(\s+NULL)/$1#Masked#$2/
Review comment:
Nit: maybe add a comment here about what's going on, to save someone
time down the road
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 445974)
Time Spent: 20m (was: 10m)
> Add queue time to compaction
>
>
> Key: HIVE-23683
> URL: https://issues.apache.org/jira/browse/HIVE-23683
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> It would be good to report to the user when the transaction is initiated.
> This info can be used when considering the health status of the compaction
> system
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23387) Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse
[ https://issues.apache.org/jira/browse/HIVE-23387?focusedWorklogId=445969&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445969 ] ASF GitHub Bot logged work on HIVE-23387: - Author: ASF GitHub Bot Created on: 15/Jun/20 17:47 Start Date: 15/Jun/20 17:47 Worklog Time Spent: 10m Work Description: nrg4878 opened a new pull request #1116: URL: https://github.com/apache/hive/pull/1116 …ath (Naveen Gangam) ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HIVE-X: Fix a typo in YYY) For more details, please see https://cwiki.apache.org/confluence/display/Hive/HowToContribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445969) Remaining Estimate: 0h Time Spent: 10m > Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse > -- > > Key: HIVE-23387 > URL: https://issues.apache.org/jira/browse/HIVE-23387 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-23387.patch, HIVE-23387.patch, HIVE-23387.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For backward compatibility, initial fix returned path that was set on db. It > could have been either from managed warehouse or external depending on what > was set. There were tests relying on certain paths to be returned. This fix > is to address the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23387) Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse
[ https://issues.apache.org/jira/browse/HIVE-23387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23387: -- Labels: pull-request-available (was: ) > Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse > -- > > Key: HIVE-23387 > URL: https://issues.apache.org/jira/browse/HIVE-23387 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23387.patch, HIVE-23387.patch, HIVE-23387.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For backward compatibility, initial fix returned path that was set on db. It > could have been either from managed warehouse or external depending on what > was set. There were tests relying on certain paths to be returned. This fix > is to address the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23603) transformDatabase() should work with changes from HIVE-22995
[ https://issues.apache.org/jira/browse/HIVE-23603?focusedWorklogId=445968&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445968 ] ASF GitHub Bot logged work on HIVE-23603: - Author: ASF GitHub Bot Created on: 15/Jun/20 17:39 Start Date: 15/Jun/20 17:39 Worklog Time Spent: 10m Work Description: nrg4878 opened a new pull request #1115: URL: https://github.com/apache/hive/pull/1115 …995 (Naveen Gangam) ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HIVE-X: Fix a typo in YYY) For more details, please see https://cwiki.apache.org/confluence/display/Hive/HowToContribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445968) Remaining Estimate: 0h Time Spent: 10m > transformDatabase() should work with changes from HIVE-22995 > > > Key: HIVE-23603 > URL: https://issues.apache.org/jira/browse/HIVE-23603 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23603.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The translation layer alters the locationUri on Database based on the > capabilities of the client. Now that we have separate locations for managed > and external for database, the implementation should be adjusted to work with > both locations. locationUri could already be external location. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23603) transformDatabase() should work with changes from HIVE-22995
[ https://issues.apache.org/jira/browse/HIVE-23603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23603: -- Labels: pull-request-available (was: ) > transformDatabase() should work with changes from HIVE-22995 > > > Key: HIVE-23603 > URL: https://issues.apache.org/jira/browse/HIVE-23603 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23603.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The translation layer alters the locationUri on Database based on the > capabilities of the client. Now that we have separate locations for managed > and external for database, the implementation should be adjusted to work with > both locations. locationUri could already be external location. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23700) HiveConf static initialization fails when JAR URI is opaque
[ https://issues.apache.org/jira/browse/HIVE-23700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francisco Guerrero reassigned HIVE-23700: - Assignee: Francisco Guerrero > HiveConf static initialization fails when JAR URI is opaque > --- > > Key: HIVE-23700 > URL: https://issues.apache.org/jira/browse/HIVE-23700 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 2.3.7 >Reporter: Francisco Guerrero >Assignee: Francisco Guerrero >Priority: Minor > Attachments: HIVE-23700.1.patch > > Original Estimate: 120h > Remaining Estimate: 120h > > HiveConf static initialization fails when the jar URI is opaque, for example > when it's embedded as a fat jar in a spring boot application. Then > initialization of the HiveConf static block fails and the HiveConf class does > not get classloaded. The opaque URI in my case looks like this > _jar:file:/usr/local/server/some-service-jar.jar!/BOOT-INF/lib/hive-common-2.3.7.jar!/_ > HiveConf#findConfigFile should be able to handle `IllegalArgumentException` > when the jar `URI` provided to `File` throws the exception. > To surface this issue three conditions need to be met. > 1. hive-site.xml should not be on the classpath > 2. hive-site.xml should not be on "HIVE_CONF_DIR" > 3. hive-site.xml should not be on "HIVE_HOME" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23700) HiveConf static initialization fails when JAR URI is opaque
[ https://issues.apache.org/jira/browse/HIVE-23700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136052#comment-17136052 ] Francisco Guerrero commented on HIVE-23700: --- Attached a patch to fix the issue. The patch handles IllegalArgumentExceptions thrown by the File constructor when the jar URI is not supported. This fixes the static initialization of the HiveConf class when four conditions are met: 1. hive-site.xml is not present on the classpath 2. hive-site.xml is not present on the "HIVE_CONF_DIR" directory 3. hive-site.xml is not present on the "HIVE_HOME" directory 4. jar URI is not absolute, or is opaque, or URI scheme is null or not file, uri authority is not null, uri fragment is not null, uri query is not null and finally uri path is empty. > HiveConf static initialization fails when JAR URI is opaque > --- > > Key: HIVE-23700 > URL: https://issues.apache.org/jira/browse/HIVE-23700 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 2.3.7 >Reporter: Francisco Guerrero >Priority: Minor > Attachments: HIVE-23700.1.patch > > Original Estimate: 120h > Remaining Estimate: 120h > > HiveConf static initialization fails when the jar URI is opaque, for example > when it's embedded as a fat jar in a spring boot application. Then > initialization of the HiveConf static block fails and the HiveConf class does > not get classloaded. The opaque URI in my case looks like this > _jar:file:/usr/local/server/some-service-jar.jar!/BOOT-INF/lib/hive-common-2.3.7.jar!/_ > HiveConf#findConfigFile should be able to handle `IllegalArgumentException` > when the jar `URI` provided to `File` throws the exception. > To surface this issue three conditions need to be met. > 1. hive-site.xml should not be on the classpath > 2. hive-site.xml should not be on "HIVE_CONF_DIR" > 3. hive-site.xml should not be on "HIVE_HOME" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23700) HiveConf static initialization fails when JAR URI is opaque
[ https://issues.apache.org/jira/browse/HIVE-23700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francisco Guerrero updated HIVE-23700: -- Attachment: HIVE-23700.1.patch > HiveConf static initialization fails when JAR URI is opaque > --- > > Key: HIVE-23700 > URL: https://issues.apache.org/jira/browse/HIVE-23700 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 2.3.7 >Reporter: Francisco Guerrero >Priority: Minor > Attachments: HIVE-23700.1.patch > > Original Estimate: 120h > Remaining Estimate: 120h > > HiveConf static initialization fails when the jar URI is opaque, for example > when it's embedded as a fat jar in a spring boot application. Then > initialization of the HiveConf static block fails and the HiveConf class does > not get classloaded. The opaque URI in my case looks like this > _jar:file:/usr/local/server/some-service-jar.jar!/BOOT-INF/lib/hive-common-2.3.7.jar!/_ > HiveConf#findConfigFile should be able to handle `IllegalArgumentException` > when the jar `URI` provided to `File` throws the exception. > To surface this issue three conditions need to be met. > 1. hive-site.xml should not be on the classpath > 2. hive-site.xml should not be on "HIVE_CONF_DIR" > 3. hive-site.xml should not be on "HIVE_HOME" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22015) [CachedStore] Cache table constraints in CachedStore
[ https://issues.apache.org/jira/browse/HIVE-22015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136011#comment-17136011 ] Adesh Kumar Rao commented on HIVE-22015: [~kishendas] I have raised the PR for this but I wasn't able to tag you. Please take a look at it and also, do let me know if I need to tag others on the PR too. > [CachedStore] Cache table constraints in CachedStore > > > Key: HIVE-22015 > URL: https://issues.apache.org/jira/browse/HIVE-22015 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Dai >Assignee: Adesh Kumar Rao >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently table constraints are not cached. Hive will pull all constraints > from tables involved in query, which results multiple db reads (including > get_primary_keys, get_foreign_keys, get_unique_constraints, etc). The effort > to cache this is small as it's just another table component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23699) Cleanup HIVEQUERYRESULTFILEFORMAT handling
[ https://issues.apache.org/jira/browse/HIVE-23699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-23699: --- > Cleanup HIVEQUERYRESULTFILEFORMAT handling > -- > > Key: HIVE-23699 > URL: https://issues.apache.org/jira/browse/HIVE-23699 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > HIVEQUERYRESULTFILEFORMAT format handling has grown over the years and has > become somewhat messy code wise in SemanticAnalyzer and TaskCompiler. There > are special cases where the HIVEQUERYRESULTFILEFORMAT setting gets changed at > runtime that may cause issues if a user changes execution.engines between > queries and probably other corner cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23665) Rewrite last_value to first_value to enable streaming results
[ https://issues.apache.org/jira/browse/HIVE-23665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23665: -- Labels: pull-request-available (was: ) > Rewrite last_value to first_value to enable streaming results > - > > Key: HIVE-23665 > URL: https://issues.apache.org/jira/browse/HIVE-23665 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23665.1.patch, HIVE-23665.2.patch, > HIVE-23665.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Rewrite last_value to first_value to enable streaming results > last_value cannot be streamed because the intermediate results need to be > buffered to determine the window result till we get the last row in the > window. But if we can rewrite to first_value we can stream the results, > although the order of results will not be guaranteed (also not important) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23665) Rewrite last_value to first_value to enable streaming results
[ https://issues.apache.org/jira/browse/HIVE-23665?focusedWorklogId=445926&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445926 ] ASF GitHub Bot logged work on HIVE-23665: - Author: ASF GitHub Bot Created on: 15/Jun/20 16:39 Start Date: 15/Jun/20 16:39 Worklog Time Spent: 10m Work Description: ramesh0201 opened a new pull request #1114: URL: https://github.com/apache/hive/pull/1114 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445926) Remaining Estimate: 0h Time Spent: 10m > Rewrite last_value to first_value to enable streaming results > - > > Key: HIVE-23665 > URL: https://issues.apache.org/jira/browse/HIVE-23665 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23665.1.patch, HIVE-23665.2.patch, > HIVE-23665.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Rewrite last_value to first_value to enable streaming results > last_value cannot be streamed because the intermediate results need to be > buffered to determine the window result till we get the last row in the > window. But if we can rewrite to first_value we can stream the results, > although the order of results will not be guaranteed (also not important) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23363: -- Priority: Critical (was: Major) > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Critical > Attachments: HIVE-23363.2.patch, HIVE-23363.patch > > > Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been > retired: > [http://www.datanucleus.org/documentation/products.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135987#comment-17135987 ] David Mollitor commented on HIVE-23363: --- Plugin updated [HIVE-17879] > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.2.patch, HIVE-23363.patch > > > Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been > retired: > [http://www.datanucleus.org/documentation/products.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-19545) Enable TestCliDriver#fouter_join_ppr.q
[ https://issues.apache.org/jira/browse/HIVE-19545?focusedWorklogId=445903&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445903 ] ASF GitHub Bot logged work on HIVE-19545: - Author: ASF GitHub Bot Created on: 15/Jun/20 16:00 Start Date: 15/Jun/20 16:00 Worklog Time Spent: 10m Work Description: kgyrtkirk merged pull request #1094: URL: https://github.com/apache/hive/pull/1094 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445903) Time Spent: 20m (was: 10m) > Enable TestCliDriver#fouter_join_ppr.q > -- > > Key: HIVE-19545 > URL: https://issues.apache.org/jira/browse/HIVE-19545 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-19545) Enable TestCliDriver#fouter_join_ppr.q
[ https://issues.apache.org/jira/browse/HIVE-19545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich resolved HIVE-19545. - Fix Version/s: 4.0.0 Resolution: Fixed pushed to master. Thank you Vineet! > Enable TestCliDriver#fouter_join_ppr.q > -- > > Key: HIVE-19545 > URL: https://issues.apache.org/jira/browse/HIVE-19545 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23689) Bump Tez version to 0.9.2
[ https://issues.apache.org/jira/browse/HIVE-23689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135972#comment-17135972 ] Zoltan Haindrich commented on HIVE-23689: - +1 > Bump Tez version to 0.9.2 > - > > Key: HIVE-23689 > URL: https://issues.apache.org/jira/browse/HIVE-23689 > Project: Hive > Issue Type: Improvement >Reporter: Jagat Singh >Assignee: Jagat Singh >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23689.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Bump Tez version to 0.9.2 from 0.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22415) Upgrade to Java 11
[
https://issues.apache.org/jira/browse/HIVE-22415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135968#comment-17135968
]
David Mollitor commented on HIVE-22415:
---
[~ashutoshc]
https://github.com/datanucleus/datanucleus-core/blob/c94645110b7c25f1e11b09bb25501e28b44d2ebf/src/main/java/org/datanucleus/asm/ClassReader.java#L169-L171
{code:java|title=ClassReader.java}
if (readShort(off + 6) > Opcodes.V1_8) {
throw new IllegalArgumentException();
}
{code}
The plugin is using the datanucleaus-core library and there is a check
specially blocking anything higher than Java 1.8
> Upgrade to Java 11
> --
>
> Key: HIVE-22415
> URL: https://issues.apache.org/jira/browse/HIVE-22415
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
>
> Upgrade Hive to Java JDK 11
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22415) Upgrade to Java 11
[
https://issues.apache.org/jira/browse/HIVE-22415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135964#comment-17135964
]
David Mollitor commented on HIVE-22415:
---
{code:none}
10:55:44,068 (main) ERROR [DataNucleus.Enhancer] - Error thrown enhancing with
ASMClassEnhancer
java.lang.IllegalArgumentException
at org.datanucleus.asm.ClassReader.(ClassReader.java:170)
at org.datanucleus.asm.ClassReader.(ClassReader.java:153)
at org.datanucleus.asm.ClassReader.(ClassReader.java:424)
at
org.datanucleus.enhancer.ClassEnhancerImpl.enhance(ClassEnhancerImpl.java:679)
at
org.datanucleus.enhancer.DataNucleusEnhancer.enhanceClass(DataNucleusEnhancer.java:932)
at
org.datanucleus.enhancer.DataNucleusEnhancer.enhance(DataNucleusEnhancer.java:527)
at
org.datanucleus.enhancer.DataNucleusEnhancer.main(DataNucleusEnhancer.java:1152)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:345)
at
org.datanucleus.maven.AbstractEnhancerMojo.enhance(AbstractEnhancerMojo.java:289)
at
org.datanucleus.maven.AbstractEnhancerMojo.executeDataNucleusTool(AbstractEnhancerMojo.java:81)
at
org.datanucleus.maven.AbstractDataNucleusMojo.execute(AbstractDataNucleusMojo.java:126)
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
{code}
> Upgrade to Java 11
> --
>
> Key: HIVE-22415
> URL: https://issues.apache.org/jira/browse/HIVE-22415
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
>
> Upgrade Hive to Java JDK 11
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23697) Fix errors in the metastore upgrade script
[ https://issues.apache.org/jira/browse/HIVE-23697?focusedWorklogId=445893&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445893 ] ASF GitHub Bot logged work on HIVE-23697: - Author: ASF GitHub Bot Created on: 15/Jun/20 15:22 Start Date: 15/Jun/20 15:22 Worklog Time Spent: 10m Work Description: aasha opened a new pull request #1113: URL: https://github.com/apache/hive/pull/1113 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HIVE-X: Fix a typo in YYY) For more details, please see https://cwiki.apache.org/confluence/display/Hive/HowToContribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445893) Remaining Estimate: 0h Time Spent: 10m > Fix errors in the metastore upgrade script > -- > > Key: HIVE-23697 > URL: https://issues.apache.org/jira/browse/HIVE-23697 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23697.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Fix a missing column separator in oracle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23697) Fix errors in the metastore upgrade script
[ https://issues.apache.org/jira/browse/HIVE-23697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23697: -- Labels: pull-request-available (was: ) > Fix errors in the metastore upgrade script > -- > > Key: HIVE-23697 > URL: https://issues.apache.org/jira/browse/HIVE-23697 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23697.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Fix a missing column separator in oracle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23697) Fix errors in the metastore upgrade script
[ https://issues.apache.org/jira/browse/HIVE-23697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23697 started by Aasha Medhi. -- > Fix errors in the metastore upgrade script > -- > > Key: HIVE-23697 > URL: https://issues.apache.org/jira/browse/HIVE-23697 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23697.01.patch > > > Fix a missing column separator in oracle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23697) Fix errors in the metastore upgrade script
[ https://issues.apache.org/jira/browse/HIVE-23697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23697: --- Attachment: HIVE-23697.01.patch Status: Patch Available (was: In Progress) > Fix errors in the metastore upgrade script > -- > > Key: HIVE-23697 > URL: https://issues.apache.org/jira/browse/HIVE-23697 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23697.01.patch > > > Fix a missing column separator in oracle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23211) Fix metastore schema differences between init scripts, and upgrade scripts
[
https://issues.apache.org/jira/browse/HIVE-23211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135958#comment-17135958
]
Aasha Medhi commented on HIVE-23211:
[~b.maidics][~szita] Facing the error :
{code:java}
138/142 ALTER TABLE TXN_COMPONENTS ALTER COLUMN TC_TXNID bigint NOT NULL;
Error: The index 'TC_TXNID_INDEX' is dependent on column 'TC_TXNID'.
(state=S0001,code=5074)
Aborting command set because "force" is false and command failed: "ALTER TABLE
TXN_COMPONENTS ALTER COLUMN TC_TXNID bigint NOT NULL;"
[ERROR] 2020-06-15 19:39:17.112 [main] MetastoreSchemaTool - Upgrade FAILED!
Metastore state would be inconsistent !!
U
{code}
Will be resolved as part of HIVE-23697
> Fix metastore schema differences between init scripts, and upgrade scripts
> --
>
> Key: HIVE-23211
> URL: https://issues.apache.org/jira/browse/HIVE-23211
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23211.1.patch, HIVE-23211.2.patch,
> HIVE-23211.3.patch
>
>
> There are some differences (character encoding, defaults etc..) in metastore
> schema if we initialize using the init scripts, or upgrade using the upgrade
> scripts. The schema should be identical.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23592) Routine "makeIntPair" is Not Correct
[
https://issues.apache.org/jira/browse/HIVE-23592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135955#comment-17135955
]
Miklos Gergely commented on HIVE-23592:
---
+1
> Routine "makeIntPair" is Not Correct
>
>
> Key: HIVE-23592
> URL: https://issues.apache.org/jira/browse/HIVE-23592
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> {code:java|title=BuddyAllocator.java}
> // Utility methods used to store pairs of ints as long.
> private static long makeIntPair(int first, int second) {
> return ((long)first) << 32 | second;
> }
> private static int getFirstInt(long result) {
> return (int) (result >>> 32);
> }
> private static int getSecondInt(long result) {
> return (int) (result & ((1L << 32) - 1));
> }
> {code}
> {code:java}
> long result = LLAP.makeIntPair(Integer.MIN_VALUE, Integer.MIN_VALUE);
> if (LLAP.getFirstInt(result) != Integer.MIN_VALUE) {
>throw new Exception();
> }
> if (LLAP.getSecondInt(result) != Integer.MIN_VALUE) {
> throw new Exception();
>}
> /*
> * Exception in thread "main" java.lang.Exception
> *at org.test.TestMe.main(TestMe.java:19)
> */
> {code}
> [https://github.com/apache/hive/blob/4b670877c280b37c5776046f66d766079489b2a8/llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java#L1677]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23697) Fix errors in the metastore upgrade script
[ https://issues.apache.org/jira/browse/HIVE-23697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-23697: - Parent: HIVE-22995 Issue Type: Sub-task (was: Task) > Fix errors in the metastore upgrade script > -- > > Key: HIVE-23697 > URL: https://issues.apache.org/jira/browse/HIVE-23697 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > > Fix a missing column separator in oracle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23697) Fix errors in the metastore upgrade script
[ https://issues.apache.org/jira/browse/HIVE-23697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-23697: - Component/s: Standalone Metastore Affects Version/s: 4.0.0 Description: Fix a missing column separator in oracle. > Fix errors in the metastore upgrade script > -- > > Key: HIVE-23697 > URL: https://issues.apache.org/jira/browse/HIVE-23697 > Project: Hive > Issue Type: Task > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > > Fix a missing column separator in oracle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135945#comment-17135945
]
Aasha Medhi commented on HIVE-22995:
Will be fixing in this patch https://issues.apache.org/jira/browse/HIVE-23697
> Add support for location for managed tables on database
> ---
>
> Key: HIVE-22995
> URL: https://issues.apache.org/jira/browse/HIVE-22995
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22995.1.patch, HIVE-22995.10.patch,
> HIVE-22995.11.patch, HIVE-22995.12.patch, HIVE-22995.2.patch,
> HIVE-22995.3.patch, HIVE-22995.4.patch, HIVE-22995.5.patch,
> HIVE-22995.6.patch, HIVE-22995.7.patch, HIVE-22995.8.patch,
> HIVE-22995.9.patch, Hive Metastore Support for Tenant-based storage
> heirarchy.pdf
>
>
> I have attached the initial spec to this jira.
> Default location for database would be the external table base directory.
> Managed location can be optionally specified.
> {code}
> CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
> [COMMENT database_comment]
> [LOCATION hdfs_path]
> [MANAGEDLOCATION hdfs_path]
> [WITH DBPROPERTIES (property_name=property_value, ...)];
> ALTER (DATABASE|SCHEMA) database_name SET
> MANAGEDLOCATION
> hdfs_path;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23697) Fix errors in the metastore upgrade script
[ https://issues.apache.org/jira/browse/HIVE-23697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23697: --- Summary: Fix errors in the metastore upgrade script (was: Fix error in the mysql metastore script) > Fix errors in the metastore upgrade script > -- > > Key: HIVE-23697 > URL: https://issues.apache.org/jira/browse/HIVE-23697 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23638) Fix FindBug issues in hive-common
[ https://issues.apache.org/jira/browse/HIVE-23638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23638: -- Description: mvn -Pspotbugs -Dorg.slf4j.simpleLogger.log.org.apache.maven.plugin.surefire.SurefirePlugin=INFO -pl :hive-common test-compile com.github.spotbugs:spotbugs-maven-plugin:4.0.0:check > Fix FindBug issues in hive-common > - > > Key: HIVE-23638 > URL: https://issues.apache.org/jira/browse/HIVE-23638 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Attachments: spotbugsXml.xml > > > mvn -Pspotbugs > -Dorg.slf4j.simpleLogger.log.org.apache.maven.plugin.surefire.SurefirePlugin=INFO > -pl :hive-common test-compile > com.github.spotbugs:spotbugs-maven-plugin:4.0.0:check -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=445872&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445872
]
ASF GitHub Bot logged work on HIVE-22869:
-
Author: ASF GitHub Bot
Created on: 15/Jun/20 14:36
Start Date: 15/Jun/20 14:36
Worklog Time Spent: 10m
Work Description: zchovan commented on a change in pull request #1073:
URL: https://github.com/apache/hive/pull/1073#discussion_r440221959
##
File path:
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/ACIDBenchmarks.java
##
@@ -0,0 +1,247 @@
+package org.apache.hadoop.hive.metastore.tools;
+
+import org.apache.hadoop.hive.metastore.api.DataOperationType;
+import org.apache.hadoop.hive.metastore.api.LockComponent;
+import org.apache.hadoop.hive.metastore.api.LockRequest;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.core.LoggerContext;
+import org.apache.logging.log4j.core.config.Configuration;
+import org.apache.thrift.TException;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static
org.apache.hadoop.hive.metastore.tools.BenchmarkUtils.createManyTables;
+import static
org.apache.hadoop.hive.metastore.tools.BenchmarkUtils.dropManyTables;
+import static
org.apache.hadoop.hive.metastore.tools.Util.throwingSupplierWrapper;
+
+public class ACIDBenchmarks {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CoreContext.class);
+
+ @State(Scope.Benchmark)
+ public static class CoreContext {
+@Param("1")
+protected int howMany;
+
+@State(Scope.Thread)
+public static class ThreadState {
+ HMSClient client;
+
+ @Setup
+ public void doSetup() throws Exception {
+LOG.debug("Creating client");
+client = HMSConfig.getInstance().newClient();
+ }
+
+ @TearDown
+ public void doTearDown() throws Exception {
+client.close();
+LOG.debug("Closed a connection to metastore.");
+ }
+}
+
+@Setup
+public void setup() {
+ LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
+ Configuration ctxConfig = ctx.getConfiguration();
+
ctxConfig.getLoggerConfig(CoreContext.class.getName()).setLevel(Level.INFO);
+ ctx.updateLoggers(ctxConfig);
+}
+ }
+
+ @State(Scope.Benchmark)
+ public static class TestOpenTxn extends CoreContext {
+
+@State(Scope.Thread)
+public static class ThreadState extends CoreContext.ThreadState {
+ List openTxns = new ArrayList<>();
+
+ @TearDown
+ public void doTearDown() throws Exception {
+client.abortTxns(openTxns);
+LOG.debug("aborted all opened txns");
+ }
+
+ void addTxn(List openTxn) {
+openTxns.addAll(openTxn);
+ }
+}
+
+@Benchmark
+public void openTxn(TestOpenTxn.ThreadState state) throws TException {
+ state.addTxn(state.client.openTxn(howMany));
+ LOG.debug("opened txns, count=", howMany);
+}
+ }
+
+ @State(Scope.Benchmark)
+ public static class TestLocking extends CoreContext {
+private int nTables;
+
+@Param("0")
+private int nPartitions;
+
+private List lockComponents;
+
+@Setup
+public void setup() {
+ this.nTables = (nPartitions != 0) ? howMany / nPartitions : howMany;
+ createLockComponents();
+}
+
+@State(Scope.Thread)
+public static class ThreadState extends CoreContext.ThreadState {
+ List openTxns = new ArrayList<>();
+ long txnId;
+
+ @Setup(org.openjdk.jmh.annotations.Level.Invocation)
+ public void iterSetup() {
+txnId = executeOpenTxnAndGetTxnId(client);
+LOG.debug("opened txn, id={}", txnId);
+openTxns.add(txnId);
+ }
+
+ @TearDown
+ public void doTearDown() throws Exception {
+client.abortTxns(openTxns);
+if (BenchmarkUtils.checkTxnsCleaned(client, openTxns) == false) {
+ LOG.error("Something went wrong with the cleanup of txns");
+}
+LOG.debug("aborted all opened txns");
+ }
+}
+
+@Benchmark
+public void lock(TestLocking.ThreadState state) {
+ LOG.debug("sending lock request");
+ executeLock(state.client, state.txnId, lockComponents);
+}
+
+private void createLockComponents() {
+ lockComponents = new ArrayList<>();
+
+ for (int i = 0; i < nTables; i++) {
+for (int j = 0; j < nPartitions - (nPartitions > 1 ? 1 : 0); j++) {
+ lockComponents.add(
+
[jira] [Updated] (HIVE-21952) Hive should allow to delete serde properties too, not just add them
[ https://issues.apache.org/jira/browse/HIVE-21952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21952: -- Labels: pull-request-available (was: ) > Hive should allow to delete serde properties too, not just add them > --- > > Key: HIVE-21952 > URL: https://issues.apache.org/jira/browse/HIVE-21952 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0, 4.0.0, 2.3.5 >Reporter: Ruslan Dautkhanov >Assignee: Miklos Gergely >Priority: Critical > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Hive should allow to delete serde properties not just add/change them > We have a use case when a presence of certain serde properties > causes issues and we want to delete just that one serde property. > It's not currently possible. > Thanks. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21952) Hive should allow to delete serde properties too, not just add them
[ https://issues.apache.org/jira/browse/HIVE-21952?focusedWorklogId=445868&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445868 ] ASF GitHub Bot logged work on HIVE-21952: - Author: ASF GitHub Bot Created on: 15/Jun/20 14:24 Start Date: 15/Jun/20 14:24 Worklog Time Spent: 10m Work Description: miklosgergely opened a new pull request #1112: URL: https://github.com/apache/hive/pull/1112 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445868) Remaining Estimate: 0h Time Spent: 10m > Hive should allow to delete serde properties too, not just add them > --- > > Key: HIVE-21952 > URL: https://issues.apache.org/jira/browse/HIVE-21952 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0, 4.0.0, 2.3.5 >Reporter: Ruslan Dautkhanov >Assignee: Miklos Gergely >Priority: Critical > Time Spent: 10m > Remaining Estimate: 0h > > Hive should allow to delete serde properties not just add/change them > We have a use case when a presence of certain serde properties > causes issues and we want to delete just that one serde property. > It's not currently possible. > Thanks. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23537) RecordReader support for static row-filtering
[ https://issues.apache.org/jira/browse/HIVE-23537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23537 started by Panagiotis Garefalakis. - > RecordReader support for static row-filtering > - > > Key: HIVE-23537 > URL: https://issues.apache.org/jira/browse/HIVE-23537 > Project: Hive > Issue Type: Sub-task > Components: llap, Reader >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > ORC-577 enables row-level filtering for the ORC format while HIVE-23167 is > aiming to extend the existing compiler logic and push filters further down > the pipeline wherever possible. > In this jira we extend the HIVE Record readers to utilize the above filtering > functionality (similar to what we already do for PPD). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23167) Expression probe decode with row-level filtering
[ https://issues.apache.org/jira/browse/HIVE-23167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23167 started by Panagiotis Garefalakis. - > Expression probe decode with row-level filtering > > > Key: HIVE-23167 > URL: https://issues.apache.org/jira/browse/HIVE-23167 > Project: Hive > Issue Type: New Feature >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > > Extend HIVE-23006 rules to support static filter expressions like: > select * from sales where sold_state = 'PR'; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23698) Compiler support for row-level filtering on filterPredicates
[ https://issues.apache.org/jira/browse/HIVE-23698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23698 started by Panagiotis Garefalakis. - > Compiler support for row-level filtering on filterPredicates > > > Key: HIVE-23698 > URL: https://issues.apache.org/jira/browse/HIVE-23698 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23698) Compiler support for row-level filtering on filterPredicates
[ https://issues.apache.org/jira/browse/HIVE-23698?focusedWorklogId=445866&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445866 ] ASF GitHub Bot logged work on HIVE-23698: - Author: ASF GitHub Bot Created on: 15/Jun/20 14:15 Start Date: 15/Jun/20 14:15 Worklog Time Spent: 10m Work Description: pgaref commented on pull request #: URL: https://github.com/apache/hive/pull/#issuecomment-644161696 Thoughts here? @jcamachor @ashutoshc This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445866) Time Spent: 20m (was: 10m) > Compiler support for row-level filtering on filterPredicates > > > Key: HIVE-23698 > URL: https://issues.apache.org/jira/browse/HIVE-23698 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23698) Compiler support for row-level filtering on filterPredicates
[ https://issues.apache.org/jira/browse/HIVE-23698?focusedWorklogId=445865&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445865 ] ASF GitHub Bot logged work on HIVE-23698: - Author: ASF GitHub Bot Created on: 15/Jun/20 14:14 Start Date: 15/Jun/20 14:14 Worklog Time Spent: 10m Work Description: pgaref opened a new pull request #: URL: https://github.com/apache/hive/pull/ Similar to what we currently do for StorageHandlers, we pushdown expression for row-level filtering when the file-format supports the feature (ORC). As we can support any type of filterExpr, the operator is then removed for the pipeline. To make sure the filter columns are not prunned, they are then added to needed columns (as part of ColumnPrunerTableScanProc). Change-Id: Ia36920b44d83ce932b2020ba9fdd30a0366a8cd8 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HIVE-X: Fix a typo in YYY) For more details, please see https://cwiki.apache.org/confluence/display/Hive/HowToContribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445865) Remaining Estimate: 0h Time Spent: 10m > Compiler support for row-level filtering on filterPredicates > > > Key: HIVE-23698 > URL: https://issues.apache.org/jira/browse/HIVE-23698 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23698) Compiler support for row-level filtering on filterPredicates
[ https://issues.apache.org/jira/browse/HIVE-23698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23698: -- Labels: pull-request-available (was: ) > Compiler support for row-level filtering on filterPredicates > > > Key: HIVE-23698 > URL: https://issues.apache.org/jira/browse/HIVE-23698 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23167) Expression probe decode with row-level filtering
[ https://issues.apache.org/jira/browse/HIVE-23167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23167: -- Summary: Expression probe decode with row-level filtering (was: Compiler support for row-level filtering on static filters) > Expression probe decode with row-level filtering > > > Key: HIVE-23167 > URL: https://issues.apache.org/jira/browse/HIVE-23167 > Project: Hive > Issue Type: New Feature >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > > Extend HIVE-23006 rules to support static filter expressions like: > select * from sales where sold_state = 'PR'; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23698) Compiler support for row-level filtering on filterPredicates
[ https://issues.apache.org/jira/browse/HIVE-23698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis reassigned HIVE-23698: - > Compiler support for row-level filtering on filterPredicates > > > Key: HIVE-23698 > URL: https://issues.apache.org/jira/browse/HIVE-23698 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21952) Hive should allow to delete serde properties too, not just add them
[ https://issues.apache.org/jira/browse/HIVE-21952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21952: - Assignee: Miklos Gergely > Hive should allow to delete serde properties too, not just add them > --- > > Key: HIVE-21952 > URL: https://issues.apache.org/jira/browse/HIVE-21952 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0, 4.0.0, 2.3.5 >Reporter: Ruslan Dautkhanov >Assignee: Miklos Gergely >Priority: Critical > > Hive should allow to delete serde properties not just add/change them > We have a use case when a presence of certain serde properties > causes issues and we want to delete just that one serde property. > It's not currently possible. > Thanks. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22415) Upgrade to Java 11
[
https://issues.apache.org/jira/browse/HIVE-22415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135842#comment-17135842
]
David Mollitor commented on HIVE-22415:
---
[~ashutoshc]
{code:none|title=JDK11}
> mvn clean install
{code}
{code:none}
[INFO] --- datanucleus-maven-plugin:5.2.1:enhance (default) @
hive-standalone-metastore-server ---
log4j:WARN No appenders could be found for logger (DataNucleus.Persistence).
log4j:WARN Please initialize the log4j system properly.
DataNucleus Enhancer (version 4.1.17) for API "JDO"
Failure during enhancement of classes - see the log for details
DataNucleus Enhancer completed with an error. Please review the enhancer log
(at DEBUG level) for full details. Some classes may have been enhanced but some
caused errors
{code}
> Upgrade to Java 11
> --
>
> Key: HIVE-22415
> URL: https://issues.apache.org/jira/browse/HIVE-22415
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
>
> Upgrade Hive to Java JDK 11
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135839#comment-17135839
]
Peter Vary commented on HIVE-22995:
---
[~ngangam], [~samuelan], [~thejas]: There is some issue with the Oracle DB
script.
{code}
-- Table DBS for classes [org.apache.hadoop.hive.metastore.model.MDatabase]
CREATE TABLE DBS
(
DB_ID NUMBER NOT NULL,
"DESC" VARCHAR2(4000) NULL,
DB_LOCATION_URI VARCHAR2(4000) NOT NULL,
"NAME" VARCHAR2(128) NULL,
OWNER_NAME VARCHAR2(128) NULL,
OWNER_TYPE VARCHAR2(10) NULL,
CTLG_NAME VARCHAR2(256) DEFAULT 'hive' NOT NULL,
CREATE_TIME NUMBER (10)
DB_MANAGED_LOCATION_URI VARCHAR2(4000) NULL
);
{code}
We have a missing ',' at the end of the CREATE_TIME column:
{code}
-- Table DBS for classes [org.apache.hadoop.hive.metastore.model.MDatabase]
CREATE TABLE DBS
(
[..]
CREATE_TIME NUMBER (10),
DB_MANAGED_LOCATION_URI VARCHAR2(4000) NULL
);
{code}
> Add support for location for managed tables on database
> ---
>
> Key: HIVE-22995
> URL: https://issues.apache.org/jira/browse/HIVE-22995
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22995.1.patch, HIVE-22995.10.patch,
> HIVE-22995.11.patch, HIVE-22995.12.patch, HIVE-22995.2.patch,
> HIVE-22995.3.patch, HIVE-22995.4.patch, HIVE-22995.5.patch,
> HIVE-22995.6.patch, HIVE-22995.7.patch, HIVE-22995.8.patch,
> HIVE-22995.9.patch, Hive Metastore Support for Tenant-based storage
> heirarchy.pdf
>
>
> I have attached the initial spec to this jira.
> Default location for database would be the external table base directory.
> Managed location can be optionally specified.
> {code}
> CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
> [COMMENT database_comment]
> [LOCATION hdfs_path]
> [MANAGEDLOCATION hdfs_path]
> [WITH DBPROPERTIES (property_name=property_value, ...)];
> ALTER (DATABASE|SCHEMA) database_name SET
> MANAGEDLOCATION
> hdfs_path;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23697) Fix error in the mysql metastore script
[ https://issues.apache.org/jira/browse/HIVE-23697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23697: --- Summary: Fix error in the mysql metastore script (was: Fix comments in the mysql metastore script) > Fix error in the mysql metastore script > --- > > Key: HIVE-23697 > URL: https://issues.apache.org/jira/browse/HIVE-23697 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23696) DB Metadata and Progress column not taking the defined length
[
https://issues.apache.org/jira/browse/HIVE-23696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-23696:
--
Labels: pull-request-available (was: )
> DB Metadata and Progress column not taking the defined length
> -
>
> Key: HIVE-23696
> URL: https://issues.apache.org/jira/browse/HIVE-23696
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23696.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Caused by: org.datanucleus.exceptions.NucleusUserException: Attempt to store
> value
> "{"dbName":"testAcidTablesReplLoadBootstrapIncr_1592205875387","replicationType":"BOOTSTRAP","stagingDir":"hdfs://localhost:65158/tmp/org_apache_hadoop_hive_ql_parse_TestReplicationScenarios_245261428230295/hrepl0/dGVzdGFjaWR0YWJsZXNyZXBsbG9hZGJvb3RzdHJhcGluY3JfMTU5MjIwNTg3NTM4Nw==/0/hive","lastReplId":25}"
> in column "RM_METADATA" that has maximum length of 255. Please correct your
> data!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23696) DB Metadata and Progress column not taking the defined length
[
https://issues.apache.org/jira/browse/HIVE-23696?focusedWorklogId=445826&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445826
]
ASF GitHub Bot logged work on HIVE-23696:
-
Author: ASF GitHub Bot
Created on: 15/Jun/20 13:04
Start Date: 15/Jun/20 13:04
Worklog Time Spent: 10m
Work Description: aasha opened a new pull request #1110:
URL: https://github.com/apache/hive/pull/1110
…ength
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HIVE-X: Fix a typo in YYY)
For more details, please see
https://cwiki.apache.org/confluence/display/Hive/HowToContribute
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 445826)
Remaining Estimate: 0h
Time Spent: 10m
> DB Metadata and Progress column not taking the defined length
> -
>
> Key: HIVE-23696
> URL: https://issues.apache.org/jira/browse/HIVE-23696
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23696.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Caused by: org.datanucleus.exceptions.NucleusUserException: Attempt to store
> value
> "{"dbName":"testAcidTablesReplLoadBootstrapIncr_1592205875387","replicationType":"BOOTSTRAP","stagingDir":"hdfs://localhost:65158/tmp/org_apache_hadoop_hive_ql_parse_TestReplicationScenarios_245261428230295/hrepl0/dGVzdGFjaWR0YWJsZXNyZXBsbG9hZGJvb3RzdHJhcGluY3JfMTU5MjIwNTg3NTM4Nw==/0/hive","lastReplId":25}"
> in column "RM_METADATA" that has maximum length of 255. Please correct your
> data!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23696) DB Metadata and Progress column not taking the defined length
[
https://issues.apache.org/jira/browse/HIVE-23696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aasha Medhi updated HIVE-23696:
---
Attachment: HIVE-23696.01.patch
Status: Patch Available (was: In Progress)
> DB Metadata and Progress column not taking the defined length
> -
>
> Key: HIVE-23696
> URL: https://issues.apache.org/jira/browse/HIVE-23696
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23696.01.patch
>
>
> Caused by: org.datanucleus.exceptions.NucleusUserException: Attempt to store
> value
> "{"dbName":"testAcidTablesReplLoadBootstrapIncr_1592205875387","replicationType":"BOOTSTRAP","stagingDir":"hdfs://localhost:65158/tmp/org_apache_hadoop_hive_ql_parse_TestReplicationScenarios_245261428230295/hrepl0/dGVzdGFjaWR0YWJsZXNyZXBsbG9hZGJvb3RzdHJhcGluY3JfMTU5MjIwNTg3NTM4Nw==/0/hive","lastReplId":25}"
> in column "RM_METADATA" that has maximum length of 255. Please correct your
> data!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-23696) DB Metadata and Progress column not taking the defined length
[
https://issues.apache.org/jira/browse/HIVE-23696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-23696 started by Aasha Medhi.
--
> DB Metadata and Progress column not taking the defined length
> -
>
> Key: HIVE-23696
> URL: https://issues.apache.org/jira/browse/HIVE-23696
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
>
> Caused by: org.datanucleus.exceptions.NucleusUserException: Attempt to store
> value
> "{"dbName":"testAcidTablesReplLoadBootstrapIncr_1592205875387","replicationType":"BOOTSTRAP","stagingDir":"hdfs://localhost:65158/tmp/org_apache_hadoop_hive_ql_parse_TestReplicationScenarios_245261428230295/hrepl0/dGVzdGFjaWR0YWJsZXNyZXBsbG9hZGJvb3RzdHJhcGluY3JfMTU5MjIwNTg3NTM4Nw==/0/hive","lastReplId":25}"
> in column "RM_METADATA" that has maximum length of 255. Please correct your
> data!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23516) Store hive replication policy execution metrics in the relational DB
[ https://issues.apache.org/jira/browse/HIVE-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135830#comment-17135830 ] Aasha Medhi commented on HIVE-23516: Tracking it here : https://issues.apache.org/jira/browse/HIVE-23697 > Store hive replication policy execution metrics in the relational DB > > > Key: HIVE-23516 > URL: https://issues.apache.org/jira/browse/HIVE-23516 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23516.01.patch, HIVE-23516.02.patch, > HIVE-23516.03.patch, HIVE-23516.04.patch, HIVE-23516.05.patch, > HIVE-23516.06.patch, HIVE-23516.07.patch, HIVE-23516.08.patch, > HIVE-23516.09.patch, HIVE-23516.10.patch, HIVE-23516.11.patch, > HIVE-23516.12.patch, HIVE-23516.13.patch, HIVE-23516.14.patch, > HIVE-23516.15.patch, HIVE-23516.16.patch, HIVE-23516.17.patch, > HIVE-23516.18.patch, HIVE-23516.19.patch, HIVE-23516.20.patch, > HIVE-23516.21.patch, HIVE-23516.22.patch, HIVE-23516.23.patch, > HIVE-23516.24.patch, HIVE-23516.25.patch, HIVE-23516.26.patch, > HIVE-23516.27.patch, HIVE-23516.28.patch, HIVE-23516.29.patch, > HIVE-23516.30.patch, HIVE-23516.31.patch, HIVE-23516.32.patch, > HIVE-23516.33.patch, HIVE-23516.34.patch, Replication Metrics.pdf > > Time Spent: 9h 20m > Remaining Estimate: 0h > > Details documented in the attached doc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23697) Fix comments in the mysql metastore script
[ https://issues.apache.org/jira/browse/HIVE-23697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi reassigned HIVE-23697: -- > Fix comments in the mysql metastore script > -- > > Key: HIVE-23697 > URL: https://issues.apache.org/jira/browse/HIVE-23697 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23516) Store hive replication policy execution metrics in the relational DB
[
https://issues.apache.org/jira/browse/HIVE-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135825#comment-17135825
]
Peter Vary edited comment on HIVE-23516 at 6/15/20, 12:50 PM:
--
[~anishek], [~aasha], [~pkumarsinha]: When I was trying to run the mysql schema
create script:
*
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql#L1245
It had a problem with the following lines:
{code}
--Create table replication metrics
[..]
--Create indexes for the replication metrics table
{code}
We need to add an extra space, like:
{code}
-- Create table replication metrics
[..]
-- Create indexes for the replication metrics table
{code}
Thanks,
Peter
was (Author: pvary):
[~anishek], [~aasha], [~pkumarsinha]: When I was trying to run the mysql schema
create script:
*
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql#L1245
It had a problem with the following lines:
{code}
--Create table replication metrics
[..]
--Create indexes for the replication metrics table
{code}
We need to add an extra space, like:
{code}
-- Create table replication metrics
[..]
-- Create indexes for the replication metrics table
{code}
> Store hive replication policy execution metrics in the relational DB
>
>
> Key: HIVE-23516
> URL: https://issues.apache.org/jira/browse/HIVE-23516
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23516.01.patch, HIVE-23516.02.patch,
> HIVE-23516.03.patch, HIVE-23516.04.patch, HIVE-23516.05.patch,
> HIVE-23516.06.patch, HIVE-23516.07.patch, HIVE-23516.08.patch,
> HIVE-23516.09.patch, HIVE-23516.10.patch, HIVE-23516.11.patch,
> HIVE-23516.12.patch, HIVE-23516.13.patch, HIVE-23516.14.patch,
> HIVE-23516.15.patch, HIVE-23516.16.patch, HIVE-23516.17.patch,
> HIVE-23516.18.patch, HIVE-23516.19.patch, HIVE-23516.20.patch,
> HIVE-23516.21.patch, HIVE-23516.22.patch, HIVE-23516.23.patch,
> HIVE-23516.24.patch, HIVE-23516.25.patch, HIVE-23516.26.patch,
> HIVE-23516.27.patch, HIVE-23516.28.patch, HIVE-23516.29.patch,
> HIVE-23516.30.patch, HIVE-23516.31.patch, HIVE-23516.32.patch,
> HIVE-23516.33.patch, HIVE-23516.34.patch, Replication Metrics.pdf
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> Details documented in the attached doc
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23516) Store hive replication policy execution metrics in the relational DB
[
https://issues.apache.org/jira/browse/HIVE-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135825#comment-17135825
]
Peter Vary commented on HIVE-23516:
---
[~anishek], [~aasha], [~pkumarsinha]: When I was trying to run the mysql schema
create script:
*
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql#L1245
It had a problem with the following lines:
{code}
--Create table replication metrics
[..]
--Create indexes for the replication metrics table
{code}
We need to add an extra space, like:
{code}
-- Create table replication metrics
[..]
-- Create indexes for the replication metrics table
{code}
> Store hive replication policy execution metrics in the relational DB
>
>
> Key: HIVE-23516
> URL: https://issues.apache.org/jira/browse/HIVE-23516
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23516.01.patch, HIVE-23516.02.patch,
> HIVE-23516.03.patch, HIVE-23516.04.patch, HIVE-23516.05.patch,
> HIVE-23516.06.patch, HIVE-23516.07.patch, HIVE-23516.08.patch,
> HIVE-23516.09.patch, HIVE-23516.10.patch, HIVE-23516.11.patch,
> HIVE-23516.12.patch, HIVE-23516.13.patch, HIVE-23516.14.patch,
> HIVE-23516.15.patch, HIVE-23516.16.patch, HIVE-23516.17.patch,
> HIVE-23516.18.patch, HIVE-23516.19.patch, HIVE-23516.20.patch,
> HIVE-23516.21.patch, HIVE-23516.22.patch, HIVE-23516.23.patch,
> HIVE-23516.24.patch, HIVE-23516.25.patch, HIVE-23516.26.patch,
> HIVE-23516.27.patch, HIVE-23516.28.patch, HIVE-23516.29.patch,
> HIVE-23516.30.patch, HIVE-23516.31.patch, HIVE-23516.32.patch,
> HIVE-23516.33.patch, HIVE-23516.34.patch, Replication Metrics.pdf
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> Details documented in the attached doc
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23696) DB Metadata and Progress column not taking the defined length
[
https://issues.apache.org/jira/browse/HIVE-23696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aasha Medhi updated HIVE-23696:
---
Description: Caused by: org.datanucleus.exceptions.NucleusUserException:
Attempt to store value
"{"dbName":"testAcidTablesReplLoadBootstrapIncr_1592205875387","replicationType":"BOOTSTRAP","stagingDir":"hdfs://localhost:65158/tmp/org_apache_hadoop_hive_ql_parse_TestReplicationScenarios_245261428230295/hrepl0/dGVzdGFjaWR0YWJsZXNyZXBsbG9hZGJvb3RzdHJhcGluY3JfMTU5MjIwNTg3NTM4Nw==/0/hive","lastReplId":25}"
in column "RM_METADATA" that has maximum length of 255. Please correct your
data!
> DB Metadata and Progress column not taking the defined length
> -
>
> Key: HIVE-23696
> URL: https://issues.apache.org/jira/browse/HIVE-23696
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
>
> Caused by: org.datanucleus.exceptions.NucleusUserException: Attempt to store
> value
> "{"dbName":"testAcidTablesReplLoadBootstrapIncr_1592205875387","replicationType":"BOOTSTRAP","stagingDir":"hdfs://localhost:65158/tmp/org_apache_hadoop_hive_ql_parse_TestReplicationScenarios_245261428230295/hrepl0/dGVzdGFjaWR0YWJsZXNyZXBsbG9hZGJvb3RzdHJhcGluY3JfMTU5MjIwNTg3NTM4Nw==/0/hive","lastReplId":25}"
> in column "RM_METADATA" that has maximum length of 255. Please correct your
> data!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23696) DB Metadata and Progress column not taking the defined length
[ https://issues.apache.org/jira/browse/HIVE-23696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi reassigned HIVE-23696: -- > DB Metadata and Progress column not taking the defined length > - > > Key: HIVE-23696 > URL: https://issues.apache.org/jira/browse/HIVE-23696 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23585) Retrieve replication instance metrics details
[ https://issues.apache.org/jira/browse/HIVE-23585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23585: --- Attachment: HIVE-23585.02.patch Status: Patch Available (was: In Progress) > Retrieve replication instance metrics details > - > > Key: HIVE-23585 > URL: https://issues.apache.org/jira/browse/HIVE-23585 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23585.01.patch, HIVE-23585.02.patch, Replication > Metrics.pdf > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23585) Retrieve replication instance metrics details
[ https://issues.apache.org/jira/browse/HIVE-23585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23585: --- Status: In Progress (was: Patch Available) > Retrieve replication instance metrics details > - > > Key: HIVE-23585 > URL: https://issues.apache.org/jira/browse/HIVE-23585 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23585.01.patch, Replication Metrics.pdf > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22015) [CachedStore] Cache table constraints in CachedStore
[ https://issues.apache.org/jira/browse/HIVE-22015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135733#comment-17135733 ] Anishek Agarwal commented on HIVE-22015: The constraint is created on the from table (Primary) and not target table(foreign). For ex if there is foreign key relationship from t1 - > t2 the relation ship is stored as part of t1 definition not as part of t2, hence the event will be generated for the primary key table and DB. > [CachedStore] Cache table constraints in CachedStore > > > Key: HIVE-22015 > URL: https://issues.apache.org/jira/browse/HIVE-22015 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Dai >Assignee: Adesh Kumar Rao >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently table constraints are not cached. Hive will pull all constraints > from tables involved in query, which results multiple db reads (including > get_primary_keys, get_foreign_keys, get_unique_constraints, etc). The effort > to cache this is small as it's just another table component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23570) [CachedStore ] Prewarm HMS cache during bootstrap with ValidWriteIdList for all the tables
[ https://issues.apache.org/jira/browse/HIVE-23570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23570 started by Adesh Kumar Rao. -- > [CachedStore ] Prewarm HMS cache during bootstrap with ValidWriteIdList for > all the tables > -- > > Key: HIVE-23570 > URL: https://issues.apache.org/jira/browse/HIVE-23570 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Adesh Kumar Rao >Priority: Major > > Since we will be caching additional ValidWriteIdList for all the tables to > provide cache consistency, cache should be prewarmed during bootstrap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[ https://issues.apache.org/jira/browse/HIVE-23571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23571 started by Adesh Kumar Rao. -- > [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper > -- > > Key: HIVE-23571 > URL: https://issues.apache.org/jira/browse/HIVE-23571 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Adesh Kumar Rao >Priority: Major > > Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in > deciding whether a given read request can be served from the cache or we have > to reload it from the backing database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23695) [CachedStore] Add unique/default constraints in CachedStore
[ https://issues.apache.org/jira/browse/HIVE-23695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao reassigned HIVE-23695: -- > [CachedStore] Add unique/default constraints in CachedStore > --- > > Key: HIVE-23695 > URL: https://issues.apache.org/jira/browse/HIVE-23695 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Adesh Kumar Rao >Assignee: Adesh Kumar Rao >Priority: Major > Fix For: 4.0.0 > > > This is blocked by HIVE-23618 (notification events are not generated for > default/unique constraints, hence created a separate sub-task from > HIVE-22015). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22015) [CachedStore] Cache table constraints in CachedStore
[ https://issues.apache.org/jira/browse/HIVE-22015?focusedWorklogId=445769&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445769 ] ASF GitHub Bot logged work on HIVE-22015: - Author: ASF GitHub Bot Created on: 15/Jun/20 10:31 Start Date: 15/Jun/20 10:31 Worklog Time Spent: 10m Work Description: adesh-rao commented on pull request #1109: URL: https://github.com/apache/hive/pull/1109#issuecomment-644046296 This patch doesn't add caching for default/unique constraints as they are aren't pushed to notification logs. I have opened a jira for the same: HIVE-23618. I will add unique/default constraints as part of separate jira once HIVE-23618 is resolved. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445769) Time Spent: 20m (was: 10m) > [CachedStore] Cache table constraints in CachedStore > > > Key: HIVE-22015 > URL: https://issues.apache.org/jira/browse/HIVE-22015 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Dai >Assignee: Adesh Kumar Rao >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently table constraints are not cached. Hive will pull all constraints > from tables involved in query, which results multiple db reads (including > get_primary_keys, get_foreign_keys, get_unique_constraints, etc). The effort > to cache this is small as it's just another table component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22015) [CachedStore] Cache table constraints in CachedStore
[ https://issues.apache.org/jira/browse/HIVE-22015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-22015: -- Labels: pull-request-available (was: ) > [CachedStore] Cache table constraints in CachedStore > > > Key: HIVE-22015 > URL: https://issues.apache.org/jira/browse/HIVE-22015 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Dai >Assignee: Adesh Kumar Rao >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently table constraints are not cached. Hive will pull all constraints > from tables involved in query, which results multiple db reads (including > get_primary_keys, get_foreign_keys, get_unique_constraints, etc). The effort > to cache this is small as it's just another table component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22015) [CachedStore] Cache table constraints in CachedStore
[ https://issues.apache.org/jira/browse/HIVE-22015?focusedWorklogId=445767&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445767 ] ASF GitHub Bot logged work on HIVE-22015: - Author: ASF GitHub Bot Created on: 15/Jun/20 10:29 Start Date: 15/Jun/20 10:29 Worklog Time Spent: 10m Work Description: adesh-rao opened a new pull request #1109: URL: https://github.com/apache/hive/pull/1109 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445767) Remaining Estimate: 0h Time Spent: 10m > [CachedStore] Cache table constraints in CachedStore > > > Key: HIVE-22015 > URL: https://issues.apache.org/jira/browse/HIVE-22015 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Dai >Assignee: Adesh Kumar Rao >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently table constraints are not cached. Hive will pull all constraints > from tables involved in query, which results multiple db reads (including > get_primary_keys, get_foreign_keys, get_unique_constraints, etc). The effort > to cache this is small as it's just another table component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-22015) [CachedStore] Cache table constraints in CachedStore
[ https://issues.apache.org/jira/browse/HIVE-22015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22015 started by Adesh Kumar Rao. -- > [CachedStore] Cache table constraints in CachedStore > > > Key: HIVE-22015 > URL: https://issues.apache.org/jira/browse/HIVE-22015 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Dai >Assignee: Adesh Kumar Rao >Priority: Major > > Currently table constraints are not cached. Hive will pull all constraints > from tables involved in query, which results multiple db reads (including > get_primary_keys, get_foreign_keys, get_unique_constraints, etc). The effort > to cache this is small as it's just another table component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22015) [CachedStore] Cache table constraints in CachedStore
[ https://issues.apache.org/jira/browse/HIVE-22015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135644#comment-17135644 ] Adesh Kumar Rao commented on HIVE-22015: [~anishek] [~thejas] While working on adding foreign keys in cached store from notification logs, I realized that DBNotificationListener is trying to process ForeignKey constraint based on PKTable instead of FKTable [here|https://github.com/apache/hive/blob/48c01107cd18d80867369e5addfa1fc5b4e7f698/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java#L658]. Shouldn't it be logically added based on the foreign key db and table instead? cc [~kishendas] > [CachedStore] Cache table constraints in CachedStore > > > Key: HIVE-22015 > URL: https://issues.apache.org/jira/browse/HIVE-22015 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Dai >Assignee: Adesh Kumar Rao >Priority: Major > > Currently table constraints are not cached. Hive will pull all constraints > from tables involved in query, which results multiple db reads (including > get_primary_keys, get_foreign_keys, get_unique_constraints, etc). The effort > to cache this is small as it's just another table component. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135642#comment-17135642 ] Zhihua Deng commented on HIVE-23633: Thank you so much for the help and review, [~kgyrtkirk]! > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23495) AcidUtils.getAcidState cleanup
[ https://issues.apache.org/jira/browse/HIVE-23495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23495: - Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master. Reviews on Review Board. Thanks for the patch, [~pvargacl]! > AcidUtils.getAcidState cleanup > -- > > Key: HIVE-23495 > URL: https://issues.apache.org/jira/browse/HIVE-23495 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23495.1.patch, HIVE-23495.2.patch, > HIVE-23495.3.patch, HIVE-23495.4.patch > > Time Spent: 20m > Remaining Estimate: 0h > > since HIVE-21225 there are two redundant implementation of the > AcidUtils.getAcidState. > The previous implementation (without the recursive listing) can be removed. > Also the performance can be improved, by removing unnecessary fileStatus > calls. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23495) AcidUtils.getAcidState cleanup
[ https://issues.apache.org/jira/browse/HIVE-23495?focusedWorklogId=445748&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445748 ] ASF GitHub Bot logged work on HIVE-23495: - Author: ASF GitHub Bot Created on: 15/Jun/20 09:07 Start Date: 15/Jun/20 09:07 Worklog Time Spent: 10m Work Description: klcopp merged pull request #1097: URL: https://github.com/apache/hive/pull/1097 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445748) Time Spent: 20m (was: 10m) > AcidUtils.getAcidState cleanup > -- > > Key: HIVE-23495 > URL: https://issues.apache.org/jira/browse/HIVE-23495 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23495.1.patch, HIVE-23495.2.patch, > HIVE-23495.3.patch, HIVE-23495.4.patch > > Time Spent: 20m > Remaining Estimate: 0h > > since HIVE-21225 there are two redundant implementation of the > AcidUtils.getAcidState. > The previous implementation (without the recursive listing) can be removed. > Also the performance can be improved, by removing unnecessary fileStatus > calls. -- This message was sent by Atlassian Jira (v8.3.4#803005)
