[jira] [Updated] (HIVE-26644) Introduce auto sizing in HMS
[ https://issues.apache.org/jira/browse/HIVE-26644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-26644: - Labels: hive-4.0.0-must pull-request-available (was: pull-request-available) > Introduce auto sizing in HMS > > > Key: HIVE-26644 > URL: https://issues.apache.org/jira/browse/HIVE-26644 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Blocker > Labels: hive-4.0.0-must, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HMS should have some ability to auto-size itself based on enabled features. > Server thread pool sizes-to-HMS connection pool sizes, larger pool sizes on > compaction-disabled-instances for better performance etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26437) dump unpartitioned Tables in parallel
[
https://issues.apache.org/jira/browse/HIVE-26437?focusedWorklogId=818680=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818680
]
ASF GitHub Bot logged work on HIVE-26437:
-
Author: ASF GitHub Bot
Created on: 20/Oct/22 05:53
Start Date: 20/Oct/22 05:53
Worklog Time Spent: 10m
Work Description: pudidic commented on code in PR #3644:
URL: https://github.com/apache/hive/pull/3644#discussion_r1000168804
##
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
##
@@ -1254,6 +1273,7 @@ Long bootStrapDump(Path dumpRoot, DumpMetaData dmd, Path
cmRoot, Hive hiveDb)
// Get the counter to store the snapshots created & deleted at
source.
replSnapshotCount = new SnapshotUtils.ReplSnapshotCount();
}
+ ExportService exportService = new ExportService(conf);
Review Comment:
Please move the ExportService object creation higher than a database level.
##
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
##
@@ -1290,6 +1310,21 @@ Long bootStrapDump(Path dumpRoot, DumpMetaData dmd, Path
cmRoot, Hive hiveDb)
}
}
+ if (exportService != null && exportService.isExportServiceRunning())
{
Review Comment:
Please move the ExportService object creation higher than a database level.
##
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
##
@@ -952,6 +954,7 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData
dmd, Path cmRoot, Hive
}
}
for(String matchedDbName : Utils.matchesDb(hiveDb,
work.dbNameOrPattern)) {
+ ExportService exportService = new ExportService(conf);
Review Comment:
Please move the ExportService object creation higher than a database level.
##
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
##
@@ -978,6 +981,22 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData
dmd, Path cmRoot, Hive
LOG.debug(te.getMessage());
}
}
+
+ if (exportService != null && exportService.isExportServiceRunning())
{
Review Comment:
Please move the ExportService object creation higher than a database level.
Issue Time Tracking
---
Worklog Id: (was: 818680)
Time Spent: 1h 40m (was: 1.5h)
> dump unpartitioned Tables in parallel
> -
>
> Key: HIVE-26437
> URL: https://issues.apache.org/jira/browse/HIVE-26437
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Amit Saonerkar
>Assignee: Amit Saonerkar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26644) Introduce auto sizing in HMS
[ https://issues.apache.org/jira/browse/HIVE-26644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-26644: - Target Version/s: 4.0.0 > Introduce auto sizing in HMS > > > Key: HIVE-26644 > URL: https://issues.apache.org/jira/browse/HIVE-26644 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Blocker > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HMS should have some ability to auto-size itself based on enabled features. > Server thread pool sizes-to-HMS connection pool sizes, larger pool sizes on > compaction-disabled-instances for better performance etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26644) Introduce auto sizing in HMS
[ https://issues.apache.org/jira/browse/HIVE-26644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-26644: - Priority: Blocker (was: Major) > Introduce auto sizing in HMS > > > Key: HIVE-26644 > URL: https://issues.apache.org/jira/browse/HIVE-26644 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Blocker > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HMS should have some ability to auto-size itself based on enabled features. > Server thread pool sizes-to-HMS connection pool sizes, larger pool sizes on > compaction-disabled-instances for better performance etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26045) Detect timed out connections for providers and auto-reconnect
[
https://issues.apache.org/jira/browse/HIVE-26045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam resolved HIVE-26045.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Fix has been merged to master. Thank you for the contribution [~zhangbutao].
> Detect timed out connections for providers and auto-reconnect
> -
>
> Key: HIVE-26045
> URL: https://issues.apache.org/jira/browse/HIVE-26045
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> For the connectors, we use single connection, no pooling. But when the
> connection is idle for an extended period, the JDBC connection times out. We
> need to check for closed connections (Connection.isClosed()?) and
> re-establish the connection. Otherwise it renders the connector fairly
> useless.
> {noformat}
> 2022-03-17T13:02:16,635 WARN [HiveServer2-Handler-Pool: Thread-116]
> thrift.ThriftCLIService: Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: SemanticException Unable to fetch table temp_dbs. Error
> retrieving remote
> table:com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
> No operations allowed after connection closed.
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:373)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:211)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:265)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:285)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:576)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:562)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_231]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_231]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_231]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_231]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at com.sun.proxy.$Proxy44.executeStatementAsync(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:567)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1550)
> ~[hive-exec-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1530)
> ~[hive-exec-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
>
[jira] [Work logged] (HIVE-26045) Detect timed out connections for providers and auto-reconnect
[
https://issues.apache.org/jira/browse/HIVE-26045?focusedWorklogId=818679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818679
]
ASF GitHub Bot logged work on HIVE-26045:
-
Author: ASF GitHub Bot
Created on: 20/Oct/22 05:28
Start Date: 20/Oct/22 05:28
Worklog Time Spent: 10m
Work Description: nrg4878 merged PR #3595:
URL: https://github.com/apache/hive/pull/3595
Issue Time Tracking
---
Worklog Id: (was: 818679)
Time Spent: 7h 10m (was: 7h)
> Detect timed out connections for providers and auto-reconnect
> -
>
> Key: HIVE-26045
> URL: https://issues.apache.org/jira/browse/HIVE-26045
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> For the connectors, we use single connection, no pooling. But when the
> connection is idle for an extended period, the JDBC connection times out. We
> need to check for closed connections (Connection.isClosed()?) and
> re-establish the connection. Otherwise it renders the connector fairly
> useless.
> {noformat}
> 2022-03-17T13:02:16,635 WARN [HiveServer2-Handler-Pool: Thread-116]
> thrift.ThriftCLIService: Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: SemanticException Unable to fetch table temp_dbs. Error
> retrieving remote
> table:com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
> No operations allowed after connection closed.
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:373)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:211)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:265)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:285)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:576)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:562)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_231]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_231]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_231]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_231]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at com.sun.proxy.$Proxy44.executeStatementAsync(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:567)
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1550)
> ~[hive-exec-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
> at
>
[jira] [Work logged] (HIVE-13353) SHOW COMPACTIONS should support filtering options
[ https://issues.apache.org/jira/browse/HIVE-13353?focusedWorklogId=818676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818676 ] ASF GitHub Bot logged work on HIVE-13353: - Author: ASF GitHub Bot Created on: 20/Oct/22 05:01 Start Date: 20/Oct/22 05:01 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3608: URL: https://github.com/apache/hive/pull/3608#issuecomment-1284927094 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3608) [](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3608=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3608=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3608=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3608=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3608=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 818676) Time Spent: 5h 10m (was: 5h) > SHOW COMPACTIONS should support filtering options > - > > Key: HIVE-13353 > URL: https://issues.apache.org/jira/browse/HIVE-13353 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.3.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: KIRTI RUGE >Priority: Major > Labels: pull-request-available > Attachments: HIVE-13353.01.patch > > Time Spent: 5h 10m > Remaining Estimate: 0h > > Since we now have historical information in SHOW COMPACTIONS the output can > easily become unwieldy. (e.g. 1000 partitions with 3 lines of history each) > this is a significant usability issue > Need to add ability to filter by db/table/partition > Perhaps would also be useful to filter by status -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26633) Make thrift max message size configurable
[
https://issues.apache.org/jira/browse/HIVE-26633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng resolved HIVE-26633.
Fix Version/s: 4.0.0
Resolution: Fixed
Merged into master. Thank you [~jfs] for the contribution and [~amansinha] for
the review!
> Make thrift max message size configurable
> -
>
> Key: HIVE-26633
> URL: https://issues.apache.org/jira/browse/HIVE-26633
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Since thrift >= 0.14, thrift now enforces max message sizes through a
> TConfiguration object as described here:
> [https://github.com/apache/thrift/blob/master/doc/specs/thrift-tconfiguration.md]
> By default MaxMessageSize gets set to 100MB.
> As a result it is possible for HMS clients not to be able to retrieve certain
> metadata for tables with a large amount of partitions or other metadata.
> For example on a cluster configured with kerberos between hs2 and hms,
> querying a large table (10k partitions, 200 columns with names of 200
> characters) results in this backtrace:
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>
> at
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
>
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:329)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:273)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:461)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:454)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema.read(FieldSchema.java:388)
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1269)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1248)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor.read(StorageDescriptor.java:1110)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1270)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
>
> at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:420)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:399)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult.read(PartitionsByExprResult.java:335)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
> {code}
> Making this configurable (and defaulting to a higher value) would allow these
> tables to still be accessible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26633) Make thrift max message size configurable
[
https://issues.apache.org/jira/browse/HIVE-26633?focusedWorklogId=818668=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818668
]
ASF GitHub Bot logged work on HIVE-26633:
-
Author: ASF GitHub Bot
Created on: 20/Oct/22 02:56
Start Date: 20/Oct/22 02:56
Worklog Time Spent: 10m
Work Description: dengzhhu653 merged PR #3674:
URL: https://github.com/apache/hive/pull/3674
Issue Time Tracking
---
Worklog Id: (was: 818668)
Time Spent: 4h 50m (was: 4h 40m)
> Make thrift max message size configurable
> -
>
> Key: HIVE-26633
> URL: https://issues.apache.org/jira/browse/HIVE-26633
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Since thrift >= 0.14, thrift now enforces max message sizes through a
> TConfiguration object as described here:
> [https://github.com/apache/thrift/blob/master/doc/specs/thrift-tconfiguration.md]
> By default MaxMessageSize gets set to 100MB.
> As a result it is possible for HMS clients not to be able to retrieve certain
> metadata for tables with a large amount of partitions or other metadata.
> For example on a cluster configured with kerberos between hs2 and hms,
> querying a large table (10k partitions, 200 columns with names of 200
> characters) results in this backtrace:
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>
> at
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
>
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:329)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:273)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:461)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:454)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema.read(FieldSchema.java:388)
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1269)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1248)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor.read(StorageDescriptor.java:1110)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1270)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
>
> at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:420)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:399)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult.read(PartitionsByExprResult.java:335)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
> {code}
> Making this configurable (and defaulting to a higher value) would allow these
> tables to still be accessible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26633) Make thrift max message size configurable
[
https://issues.apache.org/jira/browse/HIVE-26633?focusedWorklogId=818660=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818660
]
ASF GitHub Bot logged work on HIVE-26633:
-
Author: ASF GitHub Bot
Created on: 20/Oct/22 01:35
Start Date: 20/Oct/22 01:35
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3674:
URL: https://github.com/apache/hive/pull/3674#issuecomment-1284786939
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3674)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3674=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3674=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3674=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=CODE_SMELL)
[1 Code
Smell](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3674=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3674=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 818660)
Time Spent: 4h 40m (was: 4.5h)
> Make thrift max message size configurable
> -
>
> Key: HIVE-26633
> URL: https://issues.apache.org/jira/browse/HIVE-26633
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Since thrift >= 0.14, thrift now enforces max message sizes through a
> TConfiguration object as described here:
> [https://github.com/apache/thrift/blob/master/doc/specs/thrift-tconfiguration.md]
> By default MaxMessageSize gets set to 100MB.
> As a result it is possible for HMS clients not to be able to retrieve certain
> metadata for tables with a large amount of partitions or other metadata.
> For example on a cluster configured with kerberos between hs2 and hms,
> querying a large table (10k partitions, 200 columns with names of 200
> characters) results in this backtrace:
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>
> at
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>
> at
[jira] [Work logged] (HIVE-26509) Introduce dynamic leader election in HMS
[
https://issues.apache.org/jira/browse/HIVE-26509?focusedWorklogId=818659=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818659
]
ASF GitHub Bot logged work on HIVE-26509:
-
Author: ASF GitHub Bot
Created on: 20/Oct/22 01:09
Start Date: 20/Oct/22 01:09
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #3567:
URL: https://github.com/apache/hive/pull/3567#discussion_r152367
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/AuditLeaderListener.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.leader;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.TableName;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.HiveMetaStore;
+import org.apache.hadoop.hive.metastore.IHMSHandler;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.client.builder.TableBuilder;
+import org.apache.hadoop.hive.metastore.utils.SecurityUtils;
+
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.nio.charset.StandardCharsets;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+
+import static java.util.Objects.requireNonNull;
+
+public class AuditLeaderListener implements
LeaderElection.LeadershipStateListener {
+ private final Configuration configuration;
+
+ private final Path tableLocation;
+
+ private final static String SERDE =
"org.apache.hadoop.hive.serde2.JsonSerDe";
+ private final static String INPUTFORMAT =
"org.apache.hadoop.mapred.TextInputFormat";
+ private final static String OUTPUTFORMAT =
"org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat";
+
+ public AuditLeaderListener(TableName tableName, IHMSHandler handler) throws
Exception {
+requireNonNull(tableName, "tableName is null");
+requireNonNull(handler, "handler is null");
+this.configuration = handler.getConf();
+try {
+ // Store the leader info as json + text for human-readable
+ Table table = new TableBuilder()
+ .setCatName(tableName.getCat())
Review Comment:
It's initialized by:
https://github.com/apache/hive/blob/c34af3f81559c468979f20190cbd0d7f66a62b69/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java#L91-L95
and registered to the leaders by:
https://github.com/apache/hive/blob/c34af3f81559c468979f20190cbd0d7f66a62b69/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java#L113-L115
Issue Time Tracking
---
Worklog Id: (was: 818659)
Time Spent: 5h 10m (was: 5h)
> Introduce dynamic leader election in HMS
>
>
> Key: HIVE-26509
> URL: https://issues.apache.org/jira/browse/HIVE-26509
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> From HIVE-21841 we have a leader HMS selected by configuring
> metastore.housekeeping.leader.hostname on startup. This approach saves us
> from running duplicated HMS's housekeeping tasks cluster-wide.
> In this jira, we introduce another dynamic leader election: adopt hive lock
> to implement the leader election. Once a HMS owns the lock, then it becomes
> the leader, carries out the housekeeping tasks, and sends heartbeats to renew
> the lock before timeout. If the
[jira] [Work logged] (HIVE-26633) Make thrift max message size configurable
[
https://issues.apache.org/jira/browse/HIVE-26633?focusedWorklogId=818658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818658
]
ASF GitHub Bot logged work on HIVE-26633:
-
Author: ASF GitHub Bot
Created on: 20/Oct/22 00:50
Start Date: 20/Oct/22 00:50
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on PR #3674:
URL: https://github.com/apache/hive/pull/3674#issuecomment-1284747802
> The CI failure is spurious. It is the flaky oracle metastore test in which
it times out waiting for docker. It passed in the prior CI run with the same
bits (10 and 11 are on the same patchset).
>
> I think this is committable, but I can re-trigger if you want me to
@dengzhhu653
Hi @jfsii, I've re-triggered, I will merge this once it gets a green run.
Thank you!
Issue Time Tracking
---
Worklog Id: (was: 818658)
Time Spent: 4.5h (was: 4h 20m)
> Make thrift max message size configurable
> -
>
> Key: HIVE-26633
> URL: https://issues.apache.org/jira/browse/HIVE-26633
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Since thrift >= 0.14, thrift now enforces max message sizes through a
> TConfiguration object as described here:
> [https://github.com/apache/thrift/blob/master/doc/specs/thrift-tconfiguration.md]
> By default MaxMessageSize gets set to 100MB.
> As a result it is possible for HMS clients not to be able to retrieve certain
> metadata for tables with a large amount of partitions or other metadata.
> For example on a cluster configured with kerberos between hs2 and hms,
> querying a large table (10k partitions, 200 columns with names of 200
> characters) results in this backtrace:
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>
> at
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
>
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:329)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:273)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:461)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:454)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema.read(FieldSchema.java:388)
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1269)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1248)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor.read(StorageDescriptor.java:1110)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1270)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
>
> at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:420)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:399)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult.read(PartitionsByExprResult.java:335)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
> {code}
> Making this configurable (and defaulting to a higher value) would allow these
> tables to still be accessible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26621) TPC-DS query 2 fails with java.lang.RuntimeException: cannot find field _col0 from []
[
https://issues.apache.org/jira/browse/HIVE-26621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620656#comment-17620656
]
Sungwoo commented on HIVE-26621:
This is caused by HIVE-24231. By setting hive.optimize.shared.work.dppunion to
false, the optimization in HIVE-24231 becomes ineffective.
> TPC-DS query 2 fails with java.lang.RuntimeException: cannot find field _col0
> from []
> -
>
> Key: HIVE-26621
> URL: https://issues.apache.org/jira/browse/HIVE-26621
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0-alpha-1
>Reporter: Sungwoo
>Priority: Major
>
> How to reproduce:
> 1. Build Hive using the master branch. Tez version is 0.10.2.
> 2. Generate 100GB TPC-DS text data.
> 3. Load the TPC-DS data using ORC format.
> 4. Run TPC-DS queries. Query 2 fails with the error shown below.
>
> {code:java}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: cannot find field _col0 from []
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:384)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:370)
> ... 19 more
> Caused by: java.lang.RuntimeException: cannot find field _col0 from []
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:550)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:56)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:1073)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:1099)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:305)
> ... 22 more
>
> {code}
>
> A slightly simplified query that produces the same error is:
> {code:java}
> with wscs as
> (select sold_date_sk
> ,sales_price
> from (select ws_sold_date_sk sold_date_sk
> ,ws_ext_sales_price sales_price
> from web_sales) x
> union all
> (select cs_sold_date_sk sold_date_sk
> ,cs_ext_sales_price sales_price
> from catalog_sales)),
> wswscs as
> (select d_week_seq
> from wscs
> ,date_dim
> where d_date_sk = sold_date_sk
> group by d_week_seq)
> select d_week_seq1
> from
> (select wswscs.d_week_seq d_week_seq1
> from wswscs,date_dim
> where date_dim.d_week_seq = wswscs.d_week_seq and d_year = 2001) y,
> (select wswscs.d_week_seq d_week_seq2
> from wswscs,date_dim
> where date_dim.d_week_seq = wswscs.d_week_seq and d_year = 2001+1) z
> where d_week_seq1=d_week_seq2-53
> order by d_week_seq1;{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26436) Hive on MR NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-26436?focusedWorklogId=818649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818649 ] ASF GitHub Bot logged work on HIVE-26436: - Author: ASF GitHub Bot Created on: 20/Oct/22 00:29 Start Date: 20/Oct/22 00:29 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #3485: HIVE-26436: Hive on MR NullPointerException When initializeOp has not… URL: https://github.com/apache/hive/pull/3485 Issue Time Tracking --- Worklog Id: (was: 818649) Time Spent: 1h 50m (was: 1h 40m) > Hive on MR NullPointerException > --- > > Key: HIVE-26436 > URL: https://issues.apache.org/jira/browse/HIVE-26436 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 3.1.2 >Reporter: TE >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics > report from attempt_1658508427196_17228_m_00_3: Error: > java.lang.RuntimeException: Hive Runtime Error while closing operators > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) > ... 9 more > > Hive on MR run failed and the error details is above. > Hive version: 3.1.2 > > When initializeOp has not been called and close called, this exception will > happen. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26509) Introduce dynamic leader election in HMS
[
https://issues.apache.org/jira/browse/HIVE-26509?focusedWorklogId=818591=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818591
]
ASF GitHub Bot logged work on HIVE-26509:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 20:00
Start Date: 19/Oct/22 20:00
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera commented on code in PR #3567:
URL: https://github.com/apache/hive/pull/3567#discussion_r999881750
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/AuditLeaderListener.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.leader;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.TableName;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.HiveMetaStore;
+import org.apache.hadoop.hive.metastore.IHMSHandler;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.client.builder.TableBuilder;
+import org.apache.hadoop.hive.metastore.utils.SecurityUtils;
+
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.nio.charset.StandardCharsets;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+
+import static java.util.Objects.requireNonNull;
+
+public class AuditLeaderListener implements
LeaderElection.LeadershipStateListener {
+ private final Configuration configuration;
+
+ private final Path tableLocation;
+
+ private final static String SERDE =
"org.apache.hadoop.hive.serde2.JsonSerDe";
+ private final static String INPUTFORMAT =
"org.apache.hadoop.mapred.TextInputFormat";
+ private final static String OUTPUTFORMAT =
"org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat";
+
+ public AuditLeaderListener(TableName tableName, IHMSHandler handler) throws
Exception {
+requireNonNull(tableName, "tableName is null");
+requireNonNull(handler, "handler is null");
+this.configuration = handler.getConf();
+try {
+ // Store the leader info as json + text for human-readable
+ Table table = new TableBuilder()
+ .setCatName(tableName.getCat())
Review Comment:
Where is this audit leader listener called? Just want to understand if
catalog will be set in the TableName or if we need to set it if catlogname
is null.
Issue Time Tracking
---
Worklog Id: (was: 818591)
Time Spent: 5h (was: 4h 50m)
> Introduce dynamic leader election in HMS
>
>
> Key: HIVE-26509
> URL: https://issues.apache.org/jira/browse/HIVE-26509
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
> From HIVE-21841 we have a leader HMS selected by configuring
> metastore.housekeeping.leader.hostname on startup. This approach saves us
> from running duplicated HMS's housekeeping tasks cluster-wide.
> In this jira, we introduce another dynamic leader election: adopt hive lock
> to implement the leader election. Once a HMS owns the lock, then it becomes
> the leader, carries out the housekeeping tasks, and sends heartbeats to renew
> the lock before timeout. If the leader fails to reclaim the lock, then stops
> the already started tasks if it has, the electing event is audited. We can
> achieve a more dynamic leader when the original goes down or in the public
> cloud without well configured property, and reduce the leader’s burdens by
> running these tasks among
[jira] [Work logged] (HIVE-26651) MultiDelimitSerDe shouldn't rely on default charset when returning the deserialized string
[ https://issues.apache.org/jira/browse/HIVE-26651?focusedWorklogId=818579=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818579 ] ASF GitHub Bot logged work on HIVE-26651: - Author: ASF GitHub Bot Created on: 19/Oct/22 18:58 Start Date: 19/Oct/22 18:58 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3690: URL: https://github.com/apache/hive/pull/3690#issuecomment-128451 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3690) [](https://sonarcloud.io/project/issues?id=apache_hive=3690=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3690=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3690=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3690=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3690=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3690=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3690=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3690=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3690=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3690=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3690=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3690=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3690=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3690=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 818579) Time Spent: 0.5h (was: 20m) > MultiDelimitSerDe shouldn't rely on default charset when returning the > deserialized string > -- > > Key: HIVE-26651 > URL: https://issues.apache.org/jira/browse/HIVE-26651 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Same fix as HIVE-26639 but on different codepath. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26652) HiveSortPullUpConstantsRule produces an invalid plan when pulling up constants for nullable fields
[
https://issues.apache.org/jira/browse/HIVE-26652?focusedWorklogId=818570=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818570
]
ASF GitHub Bot logged work on HIVE-26652:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 18:43
Start Date: 19/Oct/22 18:43
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3692:
URL: https://github.com/apache/hive/pull/3692#issuecomment-1284428906
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3692)
[](https://sonarcloud.io/project/issues?id=apache_hive=3692=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3692=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3692=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3692=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3692=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3692=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3692=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3692=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3692=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3692=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3692=false=CODE_SMELL)
[1 Code
Smell](https://sonarcloud.io/project/issues?id=apache_hive=3692=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3692=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3692=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 818570)
Time Spent: 20m (was: 10m)
> HiveSortPullUpConstantsRule produces an invalid plan when pulling up
> constants for nullable fields
> --
>
> Key: HIVE-26652
> URL: https://issues.apache.org/jira/browse/HIVE-26652
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The rule pulls up constants without checking/adjusting nullability to match
> that of the field type.
> Here is the stack-trace when a nullable type is involved:
> {code:java}
> java.lang.AssertionError: type mismatch:
> ref:
> JavaType(class java.lang.Integer)
> input:
> JavaType(int) NOT NULL at
> org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2167)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rel.core.Project.isValid(Project.java:215)
> at org.apache.calcite.rel.core.Project.(Project.java:94)
>
[jira] [Work logged] (HIVE-26647) Implement deserialization API for commit compaction event
[ https://issues.apache.org/jira/browse/HIVE-26647?focusedWorklogId=818552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818552 ] ASF GitHub Bot logged work on HIVE-26647: - Author: ASF GitHub Bot Created on: 19/Oct/22 17:46 Start Date: 19/Oct/22 17:46 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3689: URL: https://github.com/apache/hive/pull/3689#issuecomment-1284367554 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3689) [](https://sonarcloud.io/project/issues?id=apache_hive=3689=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3689=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3689=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3689=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3689=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3689=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3689=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3689=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3689=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3689=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3689=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3689=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3689=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3689=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 818552) Time Spent: 0.5h (was: 20m) > Implement deserialization API for commit compaction event > - > > Key: HIVE-26647 > URL: https://issues.apache.org/jira/browse/HIVE-26647 > Project: Hive > Issue Type: Improvement > Components: Hive, Serializers/Deserializers >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > [HIVE-24329|https://issues.apache.org/jira/browse/HIVE-24329] introduced > notification event for commit compaction (which is serializing an event and > inserting it into the notification log). We also need to implement > Deserialization API for the commit compaction event so external listeners can > read this event. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26633) Make thrift max message size configurable
[
https://issues.apache.org/jira/browse/HIVE-26633?focusedWorklogId=818529=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818529
]
ASF GitHub Bot logged work on HIVE-26633:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 16:50
Start Date: 19/Oct/22 16:50
Worklog Time Spent: 10m
Work Description: jfsii commented on PR #3674:
URL: https://github.com/apache/hive/pull/3674#issuecomment-1284302958
The CI failure is spurious. It is the flaky oracle metastore test in which
it times out waiting for docker. It passed in the prior CI run with the same
bits (10 and 11 are on the same patchset).
I think this is committable, but I can re-trigger if you want me to
@dengzhhu653
Issue Time Tracking
---
Worklog Id: (was: 818529)
Time Spent: 4h 20m (was: 4h 10m)
> Make thrift max message size configurable
> -
>
> Key: HIVE-26633
> URL: https://issues.apache.org/jira/browse/HIVE-26633
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Since thrift >= 0.14, thrift now enforces max message sizes through a
> TConfiguration object as described here:
> [https://github.com/apache/thrift/blob/master/doc/specs/thrift-tconfiguration.md]
> By default MaxMessageSize gets set to 100MB.
> As a result it is possible for HMS clients not to be able to retrieve certain
> metadata for tables with a large amount of partitions or other metadata.
> For example on a cluster configured with kerberos between hs2 and hms,
> querying a large table (10k partitions, 200 columns with names of 200
> characters) results in this backtrace:
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>
> at
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
>
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:329)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:273)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:461)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:454)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema.read(FieldSchema.java:388)
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1269)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1248)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor.read(StorageDescriptor.java:1110)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1270)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
>
> at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:420)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:399)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult.read(PartitionsByExprResult.java:335)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
> {code}
> Making this configurable (and defaulting to a higher value) would allow these
> tables to still be accessible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26612) Hive cannot read parquet files with int64 (TIMESTAMP_MILLIS)
[
https://issues.apache.org/jira/browse/HIVE-26612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620440#comment-17620440
]
Stamatis Zampetakis commented on HIVE-26612:
It is not my intention to prove that the customer is right or wrong but rather
clarify if there is a bug and where it is. When there are multiple projects
involved in a problem (in this case Spark vs Hive) it is important to
understand which side is causing the problem. If there is a change in the way
Spark writes the Parquet file then this could also be causing the exceptions
mentioned here.
The Hive Parquet documentation
(https://cwiki.apache.org/confluence/display/Hive/Parquet) is very sketchy
leaving a lot of open questions on what exactly is supported and how things are
supposed to work. This ticket as well as HIVE-23345 present the fact that Hive
cannot read a Parquet TIMESTAMP into a Hive BIGINT as a Hive bug but there were
no tests and no documentation implying that is possible. In these cases, there
is a fine line between bug and feature request.
Another reason why I wanted to know the commit which caused the breaking change
in Hive is to understand if it was intentional or not.
Running git bisect with the test case in the PR shows that the Hive commit
which broke this use-case is HIVE-21215. Note, that if the Logical type was
missing from the metadata then things would work as before without problems.
Now I have a better picture of what is happening and it seems reasonable to fix
this; I will try to have a look in the PR in the next few days.
> Hive cannot read parquet files with int64 (TIMESTAMP_MILLIS)
>
>
> Key: HIVE-26612
> URL: https://issues.apache.org/jira/browse/HIVE-26612
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Reporter: Steve Carlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> If a parquet file has a Type of "int64 eventtime (TIMESTAMP(MILLIS,true))",
> the following error is produced:
> {noformat}
> java.lang.RuntimeException: java.io.IOException:
> org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in
> block 0 in file
> file://hive/itests/qtest/target/tmp/parquet_format_ts_as_bigint/part-0/timestamp_as_bigint.parquet
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:213)
> at org.apache.hadoop.hive.ql.exec.FetchTask.execute(FetchTask.java:98)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:212)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> Caused by: java.io.IOException:
> org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in
> block 0 in file
> file://hive/itests/qtest/target/tmp/parquet_format_ts_as_bigint/part-0/timestamp_as_bigint.parquet
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:197)
> ... 55 more
> Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
> at 1 in block 0 in file
> file:/home/stamatis/Projects/Apache/hive/itests/qtest/target/tmp/parquet_format_ts_as_bigint/part-0/timestamp_as_bigint.parquet
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:255)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:87)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:89)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:771)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:562)
> ... 57 more
> Caused by: java.lang.UnsupportedOperationException:
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$10$1
> at
> org.apache.parquet.io.api.PrimitiveConverter.addLong(PrimitiveConverter.java:105)
> at
> org.apache.parquet.column.impl.ColumnReaderBase$2$4.writeValue(ColumnReaderBase.java:301)
> at
> org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:410)
> at
>
[jira] [Work logged] (HIVE-26652) HiveSortPullUpConstantsRule produces an invalid plan when pulling up constants for nullable fields
[
https://issues.apache.org/jira/browse/HIVE-26652?focusedWorklogId=818520=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818520
]
ASF GitHub Bot logged work on HIVE-26652:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 16:00
Start Date: 19/Oct/22 16:00
Worklog Time Spent: 10m
Work Description: asolimando opened a new pull request, #3692:
URL: https://github.com/apache/hive/pull/3692
### What changes were proposed in this pull request?
Fix the rules when a nullable field is involved, since rule application
produces an invalid plan and CBO fails.
### Why are the changes needed?
Correctness
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
```mvn test -o -Dtest="TestHive*Rule" -pl ql```
Issue Time Tracking
---
Worklog Id: (was: 818520)
Remaining Estimate: 0h
Time Spent: 10m
> HiveSortPullUpConstantsRule produces an invalid plan when pulling up
> constants for nullable fields
> --
>
> Key: HIVE-26652
> URL: https://issues.apache.org/jira/browse/HIVE-26652
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Fix For: 4.0.0, 4.0.0-alpha-2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The rule pulls up constants without checking/adjusting nullability to match
> that of the field type.
> Here is the stack-trace when a nullable type is involved:
> {code:java}
> java.lang.AssertionError: type mismatch:
> ref:
> JavaType(class java.lang.Integer)
> input:
> JavaType(int) NOT NULL at
> org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2167)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rel.core.Project.isValid(Project.java:215)
> at org.apache.calcite.rel.core.Project.(Project.java:94)
> at org.apache.calcite.rel.core.Project.(Project.java:100)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.(HiveProject.java:58)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.copy(HiveProject.java:106)
> at org.apache.calcite.rel.core.Project.copy(Project.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSortPullUpConstantsRule$HiveSortPullUpConstantsRuleBase.onMatch(HiveSortPullUpConstantsRule.java:195)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
> at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.TestHiveSortExchangePullUpConstantsRule.test(TestHiveSortExchangePullUpConstantsRule.java:104)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.TestHiveSortExchangePullUpConstantsRule.testNullableFields(TestHiveSortExchangePullUpConstantsRule.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.mockito.internal.runners.DefaultInternalRunner$1$1.evaluate(DefaultInternalRunner.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at
[jira] [Updated] (HIVE-26652) HiveSortPullUpConstantsRule produces an invalid plan when pulling up constants for nullable fields
[
https://issues.apache.org/jira/browse/HIVE-26652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26652:
--
Labels: pull-request-available (was: )
> HiveSortPullUpConstantsRule produces an invalid plan when pulling up
> constants for nullable fields
> --
>
> Key: HIVE-26652
> URL: https://issues.apache.org/jira/browse/HIVE-26652
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The rule pulls up constants without checking/adjusting nullability to match
> that of the field type.
> Here is the stack-trace when a nullable type is involved:
> {code:java}
> java.lang.AssertionError: type mismatch:
> ref:
> JavaType(class java.lang.Integer)
> input:
> JavaType(int) NOT NULL at
> org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2167)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rel.core.Project.isValid(Project.java:215)
> at org.apache.calcite.rel.core.Project.(Project.java:94)
> at org.apache.calcite.rel.core.Project.(Project.java:100)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.(HiveProject.java:58)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.copy(HiveProject.java:106)
> at org.apache.calcite.rel.core.Project.copy(Project.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSortPullUpConstantsRule$HiveSortPullUpConstantsRuleBase.onMatch(HiveSortPullUpConstantsRule.java:195)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
> at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.TestHiveSortExchangePullUpConstantsRule.test(TestHiveSortExchangePullUpConstantsRule.java:104)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.TestHiveSortExchangePullUpConstantsRule.testNullableFields(TestHiveSortExchangePullUpConstantsRule.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.mockito.internal.runners.DefaultInternalRunner$1$1.evaluate(DefaultInternalRunner.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
>
[jira] [Updated] (HIVE-26653) Wrong results when (map) joining multiple tables on partition column
[
https://issues.apache.org/jira/browse/HIVE-26653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-26653:
---
Attachment: hive_26653_explain_cbo.txt
hive_26653_explain.txt
> Wrong results when (map) joining multiple tables on partition column
>
>
> Key: HIVE-26653
> URL: https://issues.apache.org/jira/browse/HIVE-26653
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: hive_26653.q, hive_26653_explain.txt,
> hive_26653_explain_cbo.txt, table_a.csv, table_b.csv
>
>
> The result of the query must have exactly one row matching the date specified
> in the WHERE clause but the query returns nothing.
> {code:sql}
> CREATE TABLE table_a (`aid` string ) PARTITIONED BY (`p_dt` string)
> row format delimited fields terminated by ',' stored as textfile;
> LOAD DATA LOCAL INPATH '../../data/files/_tbla.csv' into TABLE table_a;
> CREATE TABLE table_b (`bid` string) PARTITIONED BY (`p_dt` string)
> row format delimited fields terminated by ',' stored as textfile;
> LOAD DATA LOCAL INPATH '../../data/files/_tblb.csv' into TABLE table_b;
> set hive.auto.convert.join=true;
> set hive.optimize.semijoin.conversion=false;
> SELECT a.p_dt
> FROM ((SELECT p_dt
>FROM table_b
>GROUP BY p_dt) a
> JOIN
> (SELECT p_dt
> FROM table_a
> GROUP BY p_dt) b ON a.p_dt = b.p_dt
> JOIN
> (SELECT p_dt
> FROM table_a
> GROUP BY p_dt) c ON a.p_dt = c.p_dt)
> WHERE a.p_dt = translate(cast(to_date(date_sub('2022-08-01', 1)) AS string),
> '-', '');
> {code}
> +Expected result+
> 20220731
> +Actual result+
> Empty
> To reproduce the problem the tables need to have some data. Values in aid and
> bid columns are not important. For p_dt column use one of the following
> values 20220731, 20220630.
> I will attach some sample data with which the problem can be reproduced. The
> tables look like below.
> ||aid|pdt||
> |611|20220731|
> |239|20220630|
> |...|...|
> The problem can be reproduced via qtest in current master
> (commit
> [6b05d64ce8c7161415d97a7896ea50025322e30a|https://github.com/apache/hive/commit/6b05d64ce8c7161415d97a7896ea50025322e30a])
> by running the TestMiniLlapLocalCliDriver.
> There is specific query plan (will attach shortly) for which the problem
> shows up so if the plan changes slightly the problem may not appear anymore;
> this is why we need to set explicitly hive.optimize.semijoin.conversion and
> hive.auto.convert.join to trigger the problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-26652) HiveSortPullUpConstantsRule produces an invalid plan when pulling up constants for nullable fields
[
https://issues.apache.org/jira/browse/HIVE-26652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-26652 started by Alessandro Solimando.
---
> HiveSortPullUpConstantsRule produces an invalid plan when pulling up
> constants for nullable fields
> --
>
> Key: HIVE-26652
> URL: https://issues.apache.org/jira/browse/HIVE-26652
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Fix For: 4.0.0, 4.0.0-alpha-2
>
>
> The rule pulls up constants without checking/adjusting nullability to match
> that of the field type.
> Here is the stack-trace when a nullable type is involved:
> {code:java}
> java.lang.AssertionError: type mismatch:
> ref:
> JavaType(class java.lang.Integer)
> input:
> JavaType(int) NOT NULL at
> org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2167)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rel.core.Project.isValid(Project.java:215)
> at org.apache.calcite.rel.core.Project.(Project.java:94)
> at org.apache.calcite.rel.core.Project.(Project.java:100)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.(HiveProject.java:58)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.copy(HiveProject.java:106)
> at org.apache.calcite.rel.core.Project.copy(Project.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSortPullUpConstantsRule$HiveSortPullUpConstantsRuleBase.onMatch(HiveSortPullUpConstantsRule.java:195)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
> at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.TestHiveSortExchangePullUpConstantsRule.test(TestHiveSortExchangePullUpConstantsRule.java:104)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.TestHiveSortExchangePullUpConstantsRule.testNullableFields(TestHiveSortExchangePullUpConstantsRule.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.mockito.internal.runners.DefaultInternalRunner$1$1.evaluate(DefaultInternalRunner.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.mockito.internal.runners.DefaultInternalRunner$1.run(DefaultInternalRunner.java:99)
> at
>
[jira] [Updated] (HIVE-26653) Wrong results when (map) joining multiple tables on partition column
[
https://issues.apache.org/jira/browse/HIVE-26653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-26653:
---
Attachment: hive_26653.q
> Wrong results when (map) joining multiple tables on partition column
>
>
> Key: HIVE-26653
> URL: https://issues.apache.org/jira/browse/HIVE-26653
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: hive_26653.q, table_a.csv, table_b.csv
>
>
> The result of the query must have exactly one row matching the date specified
> in the WHERE clause but the query returns nothing.
> {code:sql}
> CREATE TABLE table_a (`aid` string ) PARTITIONED BY (`p_dt` string)
> row format delimited fields terminated by ',' stored as textfile;
> LOAD DATA LOCAL INPATH '../../data/files/_tbla.csv' into TABLE table_a;
> CREATE TABLE table_b (`bid` string) PARTITIONED BY (`p_dt` string)
> row format delimited fields terminated by ',' stored as textfile;
> LOAD DATA LOCAL INPATH '../../data/files/_tblb.csv' into TABLE table_b;
> set hive.auto.convert.join=true;
> set hive.optimize.semijoin.conversion=false;
> SELECT a.p_dt
> FROM ((SELECT p_dt
>FROM table_b
>GROUP BY p_dt) a
> JOIN
> (SELECT p_dt
> FROM table_a
> GROUP BY p_dt) b ON a.p_dt = b.p_dt
> JOIN
> (SELECT p_dt
> FROM table_a
> GROUP BY p_dt) c ON a.p_dt = c.p_dt)
> WHERE a.p_dt = translate(cast(to_date(date_sub('2022-08-01', 1)) AS string),
> '-', '');
> {code}
> +Expected result+
> 20220731
> +Actual result+
> Empty
> To reproduce the problem the tables need to have some data. Values in aid and
> bid columns are not important. For p_dt column use one of the following
> values 20220731, 20220630.
> I will attach some sample data with which the problem can be reproduced. The
> tables look like below.
> ||aid|pdt||
> |611|20220731|
> |239|20220630|
> |...|...|
> The problem can be reproduced via qtest in current master
> (commit
> [6b05d64ce8c7161415d97a7896ea50025322e30a|https://github.com/apache/hive/commit/6b05d64ce8c7161415d97a7896ea50025322e30a])
> by running the TestMiniLlapLocalCliDriver.
> There is specific query plan (will attach shortly) for which the problem
> shows up so if the plan changes slightly the problem may not appear anymore;
> this is why we need to set explicitly hive.optimize.semijoin.conversion and
> hive.auto.convert.join to trigger the problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26653) Wrong results when (map) joining multiple tables on partition column
[
https://issues.apache.org/jira/browse/HIVE-26653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-26653:
---
Attachment: table_b.csv
table_a.csv
> Wrong results when (map) joining multiple tables on partition column
>
>
> Key: HIVE-26653
> URL: https://issues.apache.org/jira/browse/HIVE-26653
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: table_a.csv, table_b.csv
>
>
> The result of the query must have exactly one row matching the date specified
> in the WHERE clause but the query returns nothing.
> {code:sql}
> CREATE TABLE table_a (`aid` string ) PARTITIONED BY (`p_dt` string)
> row format delimited fields terminated by ',' stored as textfile;
> LOAD DATA LOCAL INPATH '../../data/files/_tbla.csv' into TABLE table_a;
> CREATE TABLE table_b (`bid` string) PARTITIONED BY (`p_dt` string)
> row format delimited fields terminated by ',' stored as textfile;
> LOAD DATA LOCAL INPATH '../../data/files/_tblb.csv' into TABLE table_b;
> set hive.auto.convert.join=true;
> set hive.optimize.semijoin.conversion=false;
> SELECT a.p_dt
> FROM ((SELECT p_dt
>FROM table_b
>GROUP BY p_dt) a
> JOIN
> (SELECT p_dt
> FROM table_a
> GROUP BY p_dt) b ON a.p_dt = b.p_dt
> JOIN
> (SELECT p_dt
> FROM table_a
> GROUP BY p_dt) c ON a.p_dt = c.p_dt)
> WHERE a.p_dt = translate(cast(to_date(date_sub('2022-08-01', 1)) AS string),
> '-', '');
> {code}
> +Expected result+
> 20220731
> +Actual result+
> Empty
> To reproduce the problem the tables need to have some data. Values in aid and
> bid columns are not important. For p_dt column use one of the following
> values 20220731, 20220630.
> I will attach some sample data with which the problem can be reproduced. The
> tables look like below.
> ||aid|pdt||
> |611|20220731|
> |239|20220630|
> |...|...|
> The problem can be reproduced via qtest in current master
> (commit
> [6b05d64ce8c7161415d97a7896ea50025322e30a|https://github.com/apache/hive/commit/6b05d64ce8c7161415d97a7896ea50025322e30a])
> by running the TestMiniLlapLocalCliDriver.
> There is specific query plan (will attach shortly) for which the problem
> shows up so if the plan changes slightly the problem may not appear anymore;
> this is why we need to set explicitly hive.optimize.semijoin.conversion and
> hive.auto.convert.join to trigger the problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26653) Wrong results when (map) joining multiple tables on partition column
[
https://issues.apache.org/jira/browse/HIVE-26653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis reassigned HIVE-26653:
--
> Wrong results when (map) joining multiple tables on partition column
>
>
> Key: HIVE-26653
> URL: https://issues.apache.org/jira/browse/HIVE-26653
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> The result of the query must have exactly one row matching the date specified
> in the WHERE clause but the query returns nothing.
> {code:sql}
> CREATE TABLE table_a (`aid` string ) PARTITIONED BY (`p_dt` string)
> row format delimited fields terminated by ',' stored as textfile;
> LOAD DATA LOCAL INPATH '../../data/files/_tbla.csv' into TABLE table_a;
> CREATE TABLE table_b (`bid` string) PARTITIONED BY (`p_dt` string)
> row format delimited fields terminated by ',' stored as textfile;
> LOAD DATA LOCAL INPATH '../../data/files/_tblb.csv' into TABLE table_b;
> set hive.auto.convert.join=true;
> set hive.optimize.semijoin.conversion=false;
> SELECT a.p_dt
> FROM ((SELECT p_dt
>FROM table_b
>GROUP BY p_dt) a
> JOIN
> (SELECT p_dt
> FROM table_a
> GROUP BY p_dt) b ON a.p_dt = b.p_dt
> JOIN
> (SELECT p_dt
> FROM table_a
> GROUP BY p_dt) c ON a.p_dt = c.p_dt)
> WHERE a.p_dt = translate(cast(to_date(date_sub('2022-08-01', 1)) AS string),
> '-', '');
> {code}
> +Expected result+
> 20220731
> +Actual result+
> Empty
> To reproduce the problem the tables need to have some data. Values in aid and
> bid columns are not important. For p_dt column use one of the following
> values 20220731, 20220630.
> I will attach some sample data with which the problem can be reproduced. The
> tables look like below.
> ||aid|pdt||
> |611|20220731|
> |239|20220630|
> |...|...|
> The problem can be reproduced via qtest in current master
> (commit
> [6b05d64ce8c7161415d97a7896ea50025322e30a|https://github.com/apache/hive/commit/6b05d64ce8c7161415d97a7896ea50025322e30a])
> by running the TestMiniLlapLocalCliDriver.
> There is specific query plan (will attach shortly) for which the problem
> shows up so if the plan changes slightly the problem may not appear anymore;
> this is why we need to set explicitly hive.optimize.semijoin.conversion and
> hive.auto.convert.join to trigger the problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26652) HiveSortPullUpConstantsRule produces an invalid plan when pulling up constants for nullable fields
[
https://issues.apache.org/jira/browse/HIVE-26652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando reassigned HIVE-26652:
---
> HiveSortPullUpConstantsRule produces an invalid plan when pulling up
> constants for nullable fields
> --
>
> Key: HIVE-26652
> URL: https://issues.apache.org/jira/browse/HIVE-26652
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Fix For: 4.0.0-alpha-2
>
>
> The rule pulls up constants without checking/adjusting nullability to match
> that of the field type.
> Here is the stack-trace when a nullable type is involved:
> {code:java}
> java.lang.AssertionError: type mismatch:
> ref:
> JavaType(class java.lang.Integer)
> input:
> JavaType(int) NOT NULL at
> org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2167)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rel.core.Project.isValid(Project.java:215)
> at org.apache.calcite.rel.core.Project.(Project.java:94)
> at org.apache.calcite.rel.core.Project.(Project.java:100)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.(HiveProject.java:58)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.copy(HiveProject.java:106)
> at org.apache.calcite.rel.core.Project.copy(Project.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSortPullUpConstantsRule$HiveSortPullUpConstantsRuleBase.onMatch(HiveSortPullUpConstantsRule.java:195)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
> at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.TestHiveSortExchangePullUpConstantsRule.test(TestHiveSortExchangePullUpConstantsRule.java:104)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.TestHiveSortExchangePullUpConstantsRule.testNullableFields(TestHiveSortExchangePullUpConstantsRule.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.mockito.internal.runners.DefaultInternalRunner$1$1.evaluate(DefaultInternalRunner.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.mockito.internal.runners.DefaultInternalRunner$1.run(DefaultInternalRunner.java:99)
> at
> org.mockito.internal.runners.DefaultInternalRunner.run(DefaultInternalRunner.java:105)
[jira] [Updated] (HIVE-26652) HiveSortPullUpConstantsRule produces an invalid plan when pulling up constants for nullable fields
[
https://issues.apache.org/jira/browse/HIVE-26652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated HIVE-26652:
Fix Version/s: 4.0.0
> HiveSortPullUpConstantsRule produces an invalid plan when pulling up
> constants for nullable fields
> --
>
> Key: HIVE-26652
> URL: https://issues.apache.org/jira/browse/HIVE-26652
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Fix For: 4.0.0, 4.0.0-alpha-2
>
>
> The rule pulls up constants without checking/adjusting nullability to match
> that of the field type.
> Here is the stack-trace when a nullable type is involved:
> {code:java}
> java.lang.AssertionError: type mismatch:
> ref:
> JavaType(class java.lang.Integer)
> input:
> JavaType(int) NOT NULL at
> org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2167)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125)
> at org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57)
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rel.core.Project.isValid(Project.java:215)
> at org.apache.calcite.rel.core.Project.(Project.java:94)
> at org.apache.calcite.rel.core.Project.(Project.java:100)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.(HiveProject.java:58)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.copy(HiveProject.java:106)
> at org.apache.calcite.rel.core.Project.copy(Project.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSortPullUpConstantsRule$HiveSortPullUpConstantsRuleBase.onMatch(HiveSortPullUpConstantsRule.java:195)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
> at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.TestHiveSortExchangePullUpConstantsRule.test(TestHiveSortExchangePullUpConstantsRule.java:104)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.TestHiveSortExchangePullUpConstantsRule.testNullableFields(TestHiveSortExchangePullUpConstantsRule.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.mockito.internal.runners.DefaultInternalRunner$1$1.evaluate(DefaultInternalRunner.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.mockito.internal.runners.DefaultInternalRunner$1.run(DefaultInternalRunner.java:99)
> at
>
[jira] [Work logged] (HIVE-26645) Preparing for 4.0.0 development
[ https://issues.apache.org/jira/browse/HIVE-26645?focusedWorklogId=818509=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818509 ] ASF GitHub Bot logged work on HIVE-26645: - Author: ASF GitHub Bot Created on: 19/Oct/22 14:59 Start Date: 19/Oct/22 14:59 Worklog Time Spent: 10m Work Description: deniskuzZ merged PR #3685: URL: https://github.com/apache/hive/pull/3685 Issue Time Tracking --- Worklog Id: (was: 818509) Time Spent: 1.5h (was: 1h 20m) > Preparing for 4.0.0 development > --- > > Key: HIVE-26645 > URL: https://issues.apache.org/jira/browse/HIVE-26645 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26636) Hyperbolic functions
[ https://issues.apache.org/jira/browse/HIVE-26636?focusedWorklogId=818504=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818504 ] ASF GitHub Bot logged work on HIVE-26636: - Author: ASF GitHub Bot Created on: 19/Oct/22 14:51 Start Date: 19/Oct/22 14:51 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3681: URL: https://github.com/apache/hive/pull/3681#issuecomment-1284142035 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3681) [](https://sonarcloud.io/project/issues?id=apache_hive=3681=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3681=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3681=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3681=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3681=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3681=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3681=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3681=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3681=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3681=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3681=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3681=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3681=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3681=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 818504) Time Spent: 0.5h (was: 20m) > Hyperbolic functions > -- > > Key: HIVE-26636 > URL: https://issues.apache.org/jira/browse/HIVE-26636 > Project: Hive > Issue Type: New Feature >Reporter: Gopinath >Assignee: Gopinath >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Adding hyperbolic functions. > |Function|Description|Example|Result| > |sinh (x)|hyperbolic sine|sinh(0)|0| > |cosh (x)|hyperbolic cosine|cosh(0)|1| > |tanh (x)|hyperbolic tangent|tanh(0)|0| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26645) Preparing for 4.0.0 development
[ https://issues.apache.org/jira/browse/HIVE-26645?focusedWorklogId=818501=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818501 ] ASF GitHub Bot logged work on HIVE-26645: - Author: ASF GitHub Bot Created on: 19/Oct/22 14:49 Start Date: 19/Oct/22 14:49 Worklog Time Spent: 10m Work Description: nrg4878 commented on PR #3685: URL: https://github.com/apache/hive/pull/3685#issuecomment-1284139305 @ayushtkn Can you take a quick look as well before we merge it? Issue Time Tracking --- Worklog Id: (was: 818501) Time Spent: 1h 20m (was: 1h 10m) > Preparing for 4.0.0 development > --- > > Key: HIVE-26645 > URL: https://issues.apache.org/jira/browse/HIVE-26645 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26633) Make thrift max message size configurable
[
https://issues.apache.org/jira/browse/HIVE-26633?focusedWorklogId=818484=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818484
]
ASF GitHub Bot logged work on HIVE-26633:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 13:38
Start Date: 19/Oct/22 13:38
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3674:
URL: https://github.com/apache/hive/pull/3674#issuecomment-1284033380
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3674)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3674=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3674=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3674=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=CODE_SMELL)
[1 Code
Smell](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3674=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3674=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 818484)
Time Spent: 4h 10m (was: 4h)
> Make thrift max message size configurable
> -
>
> Key: HIVE-26633
> URL: https://issues.apache.org/jira/browse/HIVE-26633
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Since thrift >= 0.14, thrift now enforces max message sizes through a
> TConfiguration object as described here:
> [https://github.com/apache/thrift/blob/master/doc/specs/thrift-tconfiguration.md]
> By default MaxMessageSize gets set to 100MB.
> As a result it is possible for HMS clients not to be able to retrieve certain
> metadata for tables with a large amount of partitions or other metadata.
> For example on a cluster configured with kerberos between hs2 and hms,
> querying a large table (10k partitions, 200 columns with names of 200
> characters) results in this backtrace:
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>
> at
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>
> at
[jira] [Work logged] (HIVE-26645) Preparing for 4.0.0 development
[ https://issues.apache.org/jira/browse/HIVE-26645?focusedWorklogId=818447=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818447 ] ASF GitHub Bot logged work on HIVE-26645: - Author: ASF GitHub Bot Created on: 19/Oct/22 12:07 Start Date: 19/Oct/22 12:07 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3685: URL: https://github.com/apache/hive/pull/3685#issuecomment-1283906773 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3685) [](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3685=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3685=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3685=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3685=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3685=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 818447) Time Spent: 1h 10m (was: 1h) > Preparing for 4.0.0 development > --- > > Key: HIVE-26645 > URL: https://issues.apache.org/jira/browse/HIVE-26645 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26555) Read-only mode for Hive database
[ https://issues.apache.org/jira/browse/HIVE-26555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620299#comment-17620299 ] László Bodor commented on HIVE-26555: - design question: can you please describe what kind of failover/fail-back scenarios are you referring to and how exactly a read-only database is a solution in that event? thanks! > Read-only mode for Hive database > > > Key: HIVE-26555 > URL: https://issues.apache.org/jira/browse/HIVE-26555 > Project: Hive > Issue Type: New Feature >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 1h > Remaining Estimate: 0h > > h1. Purpose > In failover/fail-back scenarios, a Hive database needs to be read-only, while > other one is writable to keep a single source of truth. > h1. User-Facing Changes > Yes. EnforceReadOnlyDatabaseHook class implements ExecuteWithHookContext > interface. hive.exec.pre.hooks needs to have the class name to initiate an > instance. The "readonly" database property can be configured to turn it on > and off. > h2. Allowed read operations > All read operations without any data/metadata change are allowed. > * EXPLAIN > * USE(or SWITCHDATABASE) > * REPLDUMP > * REPLSTATUS > * EXPORT > * KILL_QUERY > * DESC prefix > * SHOW prefix > * QUERY with SELECT or EXPLAIN. INSERT, DELETE, UPDATE are disallowed. > h2. Allowed write operations > Most of write operations that change data/metadata are disallowed. There are > few allowed exceptions. The first one is alter database to make a database > writable. The second one is replication load to load a dumped database. > * ALTER DATABASE db_name SET DBPROPERTIES without "readonly"="true". > * REPLLOAD > h1. Tests > * read_only_hook.q: USE, SHOW, DESC, DESCRIBE, EXPLAIN, SELECT > * read_only_delete.q > * read_only_insert.q -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26642) Replace HiveFilterMergeRule with Calcite's built-in implementation
[ https://issues.apache.org/jira/browse/HIVE-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis resolved HIVE-26642. Fix Version/s: 4.0.0 4.0.0-alpha-2 Resolution: Fixed Fixed in https://github.com/apache/hive/commit/4ae2dae62398f319f1daa0cea77edd88b6c8f004. Thanks for the review [~kkasa]! > Replace HiveFilterMergeRule with Calcite's built-in implementation > -- > > Key: HIVE-26642 > URL: https://issues.apache.org/jira/browse/HIVE-26642 > Project: Hive > Issue Type: Improvement >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 4.0.0-alpha-2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The rule was copied from Calcite to address HIVE-23389 as a temporary > workaround till the next Calcite upgrade. > Now that Hive is on calcite 1.25.0 (HIVE-23456) the in-house copy can be > removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26643) HiveUnionPullUpConstantsRule produces an invalid plan when pulling up constants for nullable fields
[
https://issues.apache.org/jira/browse/HIVE-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-26643.
Fix Version/s: 4.0.0
4.0.0-alpha-2
Resolution: Fixed
Fixed in
https://github.com/apache/hive/commit/6b05d64ce8c7161415d97a7896ea50025322e30a.
Thanks for the PR [~asolimando]!
> HiveUnionPullUpConstantsRule produces an invalid plan when pulling up
> constants for nullable fields
> ---
>
> Key: HIVE-26643
> URL: https://issues.apache.org/jira/browse/HIVE-26643
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-2
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The rule pulls up constants without checking/adjusting nullability to match
> that of the field type.
> Here is the stack-trace when a nullable type is involved:
> {code:java}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(JavaType(class java.lang.Integer) f1, JavaType(int)
> NOT NULL f2) NOT NULL
> expression type is RecordType(JavaType(int) NOT NULL f1, JavaType(int) NOT
> NULL f2) NOT NULL
> set is
> rel#38:HiveUnion.(input#0=HepRelVertex#35,input#1=HepRelVertex#35,all=true)
> expression is HiveProject(f1=[1], f2=[$0])
> HiveUnion(all=[true])
> HiveProject(f2=[$1])
> HiveProject(f1=[$0], f2=[$1])
> HiveFilter(condition=[=($0, 1)])
> LogicalTableScan(table=[[]])
> HiveProject(f2=[$1])
> HiveProject(f1=[$0], f2=[$1])
> HiveFilter(condition=[=($0, 1)])
> LogicalTableScan(table=[[]])
> {code}
> The solution is to check nullability and add a cast when the field is
> nullable, since the constant's type is not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26638) Replace in-house CBO reduce expressions rules with Calcite's built-in classes
[
https://issues.apache.org/jira/browse/HIVE-26638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-26638.
Fix Version/s: 4.0.0
4.0.0-alpha-2
Resolution: Fixed
Fixed in
https://github.com/apache/hive/commit/5a2b42982adeca506daf5bec435dfc51b4522638.
Thanks for the review [~kkasa]!
> Replace in-house CBO reduce expressions rules with Calcite's built-in classes
> -
>
> Key: HIVE-26638
> URL: https://issues.apache.org/jira/browse/HIVE-26638
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The goal of this ticket is to remove Hive specific code in
> [HiveReduceExpressionsRule|https://github.com/apache/hive/blob/b48c1bf11c4f75ba2c894e4732a96813ddde1414/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java]
> and use exclusively the respective Calcite classes (i.e.,
> [ReduceExpressionsRule|https://github.com/apache/calcite/blob/2c30a56158cdd351d35725006bc1f76bb6aac75b/core/src/main/java/org/apache/calcite/rel/rules/ReduceExpressionsRule.java])
> to reduce maintenance overhead and facilitate code evolution.
> Currently the only difference between in-house (HiveReduceExpressionsRule)
> and built-in (ReduceExpressionsRule) reduce expressions rules lies in the way
> we treat the {{Filter}} operator (i.e., FilterReduceExpressionsRule).
> There are four differences when comparing the in-house code with the
> respective part in Calcite 1.25.0 that are Hive specific.
> +Match nullability when reducing expressions+
> When we reduce filters we always set {{matchNullability}} (last parameter) to
> false.
> {code:java}
> if (reduceExpressions(filter, expList, predicates, true, false)) {
> {code}
> This means that the original and reduced expression can have a slightly
> different type in terms of nullability; the original is nullable and the
> reduced is not nullable. When the value is true the type can be preserved by
> adding a "nullability" CAST, which is a cast to the same type which differs
> only to if it is nullable or not.
> Hardcoding {{matchNullability}} to false was done as part of the upgrade in
> Calcite 1.15.0 (HIVE-18068) where the behavior of the rule became
> configurable (CALCITE-2041).
> +Remove nullability cast explicitly+
> When the expression is reduced we try to remove the nullability cast; if
> there is one.
> {code:java}
> if (RexUtil.isNullabilityCast(filter.getCluster().getTypeFactory(),
> newConditionExp)) {
> newConditionExp = ((RexCall) newConditionExp).getOperands().get(0);
> }
> {code}
> The code was added as part of the upgrade to Calcite 1.10.0 (HIVE-13316).
> However, the code is redundant as of HIVE-18068; setting {{matchNullability}}
> to {{false}} no longer generates nullability casts during the reduction.
> +Avoid creating filters with condition of type NULL+
> {code:java}
> if(newConditionExp.getType().getSqlTypeName() == SqlTypeName.NULL) {
> newConditionExp = call.builder().cast(newConditionExp,
> SqlTypeName.BOOLEAN);
> }
> {code}
> Hive tries to cast such expressions to BOOLEAN to avoid the weird (and
> possibly problematic) situation of having a condition with NULL type.
> In Calcite, there is specific code for detecting if the new condition is the
> NULL literal (with NULL type) and if that's the case it turns the relation to
> empty.
> {code:java}
> } else if (newConditionExp instanceof RexLiteral
> || RexUtil.isNullLiteral(newConditionExp, true)) {
> call.transformTo(createEmptyRelOrEquivalent(call, filter));
> {code}
> Due to that the Hive specific code is redundant if the Calcite rule is used.
> +Bail out when input to reduceNotNullableFilter is not a RexCall+
> {code:java}
> if (!(rexCall.getOperands().get(0) instanceof RexCall)) {
> // If child is not a RexCall instance, we can bail out
> return;
> }
> {code}
> The code was added as part of the upgrade to Calcite 1.10.0 (HIVE-13316) but
> it does not add any functional value.
> The instanceof check is redundant since the code in reduceNotNullableFilter
> [is a
> noop|https://github.com/apache/hive/blob/6e8fc53fb68898d1a404435859cea5bbc79200a4/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java#L228]
> when the expression/call is not one of the following: IS_NULL, IS_UNKNOWN,
> IS_NOT_NULL, which are all rex calls.
> +Summary+
> All of the Hive specific changes mentioned previously can be safely replaced
> by appropriate uses of the Calcite APIs
[jira] [Work logged] (HIVE-26638) Replace in-house CBO reduce expressions rules with Calcite's built-in classes
[
https://issues.apache.org/jira/browse/HIVE-26638?focusedWorklogId=818438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818438
]
ASF GitHub Bot logged work on HIVE-26638:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 11:53
Start Date: 19/Oct/22 11:53
Worklog Time Spent: 10m
Work Description: zabetak closed pull request #3666: HIVE-26638: Replace
in-house CBO reduce expressions rules with Calcite's built-in classes
URL: https://github.com/apache/hive/pull/3666
Issue Time Tracking
---
Worklog Id: (was: 818438)
Time Spent: 20m (was: 10m)
> Replace in-house CBO reduce expressions rules with Calcite's built-in classes
> -
>
> Key: HIVE-26638
> URL: https://issues.apache.org/jira/browse/HIVE-26638
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The goal of this ticket is to remove Hive specific code in
> [HiveReduceExpressionsRule|https://github.com/apache/hive/blob/b48c1bf11c4f75ba2c894e4732a96813ddde1414/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java]
> and use exclusively the respective Calcite classes (i.e.,
> [ReduceExpressionsRule|https://github.com/apache/calcite/blob/2c30a56158cdd351d35725006bc1f76bb6aac75b/core/src/main/java/org/apache/calcite/rel/rules/ReduceExpressionsRule.java])
> to reduce maintenance overhead and facilitate code evolution.
> Currently the only difference between in-house (HiveReduceExpressionsRule)
> and built-in (ReduceExpressionsRule) reduce expressions rules lies in the way
> we treat the {{Filter}} operator (i.e., FilterReduceExpressionsRule).
> There are four differences when comparing the in-house code with the
> respective part in Calcite 1.25.0 that are Hive specific.
> +Match nullability when reducing expressions+
> When we reduce filters we always set {{matchNullability}} (last parameter) to
> false.
> {code:java}
> if (reduceExpressions(filter, expList, predicates, true, false)) {
> {code}
> This means that the original and reduced expression can have a slightly
> different type in terms of nullability; the original is nullable and the
> reduced is not nullable. When the value is true the type can be preserved by
> adding a "nullability" CAST, which is a cast to the same type which differs
> only to if it is nullable or not.
> Hardcoding {{matchNullability}} to false was done as part of the upgrade in
> Calcite 1.15.0 (HIVE-18068) where the behavior of the rule became
> configurable (CALCITE-2041).
> +Remove nullability cast explicitly+
> When the expression is reduced we try to remove the nullability cast; if
> there is one.
> {code:java}
> if (RexUtil.isNullabilityCast(filter.getCluster().getTypeFactory(),
> newConditionExp)) {
> newConditionExp = ((RexCall) newConditionExp).getOperands().get(0);
> }
> {code}
> The code was added as part of the upgrade to Calcite 1.10.0 (HIVE-13316).
> However, the code is redundant as of HIVE-18068; setting {{matchNullability}}
> to {{false}} no longer generates nullability casts during the reduction.
> +Avoid creating filters with condition of type NULL+
> {code:java}
> if(newConditionExp.getType().getSqlTypeName() == SqlTypeName.NULL) {
> newConditionExp = call.builder().cast(newConditionExp,
> SqlTypeName.BOOLEAN);
> }
> {code}
> Hive tries to cast such expressions to BOOLEAN to avoid the weird (and
> possibly problematic) situation of having a condition with NULL type.
> In Calcite, there is specific code for detecting if the new condition is the
> NULL literal (with NULL type) and if that's the case it turns the relation to
> empty.
> {code:java}
> } else if (newConditionExp instanceof RexLiteral
> || RexUtil.isNullLiteral(newConditionExp, true)) {
> call.transformTo(createEmptyRelOrEquivalent(call, filter));
> {code}
> Due to that the Hive specific code is redundant if the Calcite rule is used.
> +Bail out when input to reduceNotNullableFilter is not a RexCall+
> {code:java}
> if (!(rexCall.getOperands().get(0) instanceof RexCall)) {
> // If child is not a RexCall instance, we can bail out
> return;
> }
> {code}
> The code was added as part of the upgrade to Calcite 1.10.0 (HIVE-13316) but
> it does not add any functional value.
> The instanceof check is redundant since the code in reduceNotNullableFilter
> [is a
> noop|https://github.com/apache/hive/blob/6e8fc53fb68898d1a404435859cea5bbc79200a4/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java#L228]
> when the
[jira] [Work logged] (HIVE-26642) Replace HiveFilterMergeRule with Calcite's built-in implementation
[ https://issues.apache.org/jira/browse/HIVE-26642?focusedWorklogId=818440=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818440 ] ASF GitHub Bot logged work on HIVE-26642: - Author: ASF GitHub Bot Created on: 19/Oct/22 11:53 Start Date: 19/Oct/22 11:53 Worklog Time Spent: 10m Work Description: zabetak closed pull request #3678: HIVE-26642: Replace HiveFilterMergeRule with Calcite's built-in implementation URL: https://github.com/apache/hive/pull/3678 Issue Time Tracking --- Worklog Id: (was: 818440) Time Spent: 0.5h (was: 20m) > Replace HiveFilterMergeRule with Calcite's built-in implementation > -- > > Key: HIVE-26642 > URL: https://issues.apache.org/jira/browse/HIVE-26642 > Project: Hive > Issue Type: Improvement >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The rule was copied from Calcite to address HIVE-23389 as a temporary > workaround till the next Calcite upgrade. > Now that Hive is on calcite 1.25.0 (HIVE-23456) the in-house copy can be > removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26643) HiveUnionPullUpConstantsRule produces an invalid plan when pulling up constants for nullable fields
[
https://issues.apache.org/jira/browse/HIVE-26643?focusedWorklogId=818439=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818439
]
ASF GitHub Bot logged work on HIVE-26643:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 11:53
Start Date: 19/Oct/22 11:53
Worklog Time Spent: 10m
Work Description: zabetak closed pull request #3680: [HIVE-26643]
HiveUnionPullUpConstantsRule produces an invalid plan when pulling up constants
for nullable fields
URL: https://github.com/apache/hive/pull/3680
Issue Time Tracking
---
Worklog Id: (was: 818439)
Time Spent: 40m (was: 0.5h)
> HiveUnionPullUpConstantsRule produces an invalid plan when pulling up
> constants for nullable fields
> ---
>
> Key: HIVE-26643
> URL: https://issues.apache.org/jira/browse/HIVE-26643
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The rule pulls up constants without checking/adjusting nullability to match
> that of the field type.
> Here is the stack-trace when a nullable type is involved:
> {code:java}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(JavaType(class java.lang.Integer) f1, JavaType(int)
> NOT NULL f2) NOT NULL
> expression type is RecordType(JavaType(int) NOT NULL f1, JavaType(int) NOT
> NULL f2) NOT NULL
> set is
> rel#38:HiveUnion.(input#0=HepRelVertex#35,input#1=HepRelVertex#35,all=true)
> expression is HiveProject(f1=[1], f2=[$0])
> HiveUnion(all=[true])
> HiveProject(f2=[$1])
> HiveProject(f1=[$0], f2=[$1])
> HiveFilter(condition=[=($0, 1)])
> LogicalTableScan(table=[[]])
> HiveProject(f2=[$1])
> HiveProject(f1=[$0], f2=[$1])
> HiveFilter(condition=[=($0, 1)])
> LogicalTableScan(table=[[]])
> {code}
> The solution is to check nullability and add a cast when the field is
> nullable, since the constant's type is not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26628) Iceberg table is created when running explain ctas command
[
https://issues.apache.org/jira/browse/HIVE-26628?focusedWorklogId=818436=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818436
]
ASF GitHub Bot logged work on HIVE-26628:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 11:49
Start Date: 19/Oct/22 11:49
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3670:
URL: https://github.com/apache/hive/pull/3670#issuecomment-1283881682
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3670)
[](https://sonarcloud.io/project/issues?id=apache_hive=3670=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3670=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3670=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3670=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3670=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3670=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3670=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3670=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3670=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3670=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3670=false=CODE_SMELL)
[4 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3670=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3670=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3670=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 818436)
Time Spent: 0.5h (was: 20m)
> Iceberg table is created when running explain ctas command
> --
>
> Key: HIVE-26628
> URL: https://issues.apache.org/jira/browse/HIVE-26628
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code}
> create table source(a int, b string, c int);
> explain
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> {code}
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Table already exists:
> default.tbl_ice
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:13963)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12528)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12693)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
> at
>
[jira] [Work logged] (HIVE-26651) MultiDelimitSerDe shouldn't rely on default charset when returning the deserialized string
[ https://issues.apache.org/jira/browse/HIVE-26651?focusedWorklogId=818431=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818431 ] ASF GitHub Bot logged work on HIVE-26651: - Author: ASF GitHub Bot Created on: 19/Oct/22 11:19 Start Date: 19/Oct/22 11:19 Worklog Time Spent: 10m Work Description: abstractdog commented on PR #3690: URL: https://github.com/apache/hive/pull/3690#issuecomment-1283847556 @ayushtkn one more, please :) thanks! Issue Time Tracking --- Worklog Id: (was: 818431) Time Spent: 20m (was: 10m) > MultiDelimitSerDe shouldn't rely on default charset when returning the > deserialized string > -- > > Key: HIVE-26651 > URL: https://issues.apache.org/jira/browse/HIVE-26651 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Same fix as HIVE-26639 but on different codepath. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26651) MultiDelimitSerDe shouldn't rely on default charset when returning the deserialized string
[ https://issues.apache.org/jira/browse/HIVE-26651?focusedWorklogId=818430=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818430 ] ASF GitHub Bot logged work on HIVE-26651: - Author: ASF GitHub Bot Created on: 19/Oct/22 11:18 Start Date: 19/Oct/22 11:18 Worklog Time Spent: 10m Work Description: abstractdog opened a new pull request, #3690: URL: https://github.com/apache/hive/pull/3690 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Issue Time Tracking --- Worklog Id: (was: 818430) Remaining Estimate: 0h Time Spent: 10m > MultiDelimitSerDe shouldn't rely on default charset when returning the > deserialized string > -- > > Key: HIVE-26651 > URL: https://issues.apache.org/jira/browse/HIVE-26651 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Same fix as HIVE-26639 but on different codepath. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26651) MultiDelimitSerDe shouldn't rely on default charset when returning the deserialized string
[ https://issues.apache.org/jira/browse/HIVE-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26651: -- Labels: pull-request-available (was: ) > MultiDelimitSerDe shouldn't rely on default charset when returning the > deserialized string > -- > > Key: HIVE-26651 > URL: https://issues.apache.org/jira/browse/HIVE-26651 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Same fix as HIVE-26639 but on different codepath. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26651) MultiDelimitSerDe shouldn't rely on default charset when returning the deserialized string
[ https://issues.apache.org/jira/browse/HIVE-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-26651: Summary: MultiDelimitSerDe shouldn't rely on default charset when returning the deserialized string (was: MultiDelimitSerDe shouldn't rely on default charset) > MultiDelimitSerDe shouldn't rely on default charset when returning the > deserialized string > -- > > Key: HIVE-26651 > URL: https://issues.apache.org/jira/browse/HIVE-26651 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > Same fix as HIVE-26639 but on different codepath. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26651) MultiDelimitSerDe shouldn't rely on default charset
[ https://issues.apache.org/jira/browse/HIVE-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned HIVE-26651: --- Assignee: László Bodor > MultiDelimitSerDe shouldn't rely on default charset > --- > > Key: HIVE-26651 > URL: https://issues.apache.org/jira/browse/HIVE-26651 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-26651) MultiDelimitSerDe shouldn't rely on default charset
[ https://issues.apache.org/jira/browse/HIVE-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-26651 started by László Bodor. --- > MultiDelimitSerDe shouldn't rely on default charset > --- > > Key: HIVE-26651 > URL: https://issues.apache.org/jira/browse/HIVE-26651 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > Same fix as HIVE-26639 but on different codepath. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26651) MultiDelimitSerDe shouldn't rely on default charset
[ https://issues.apache.org/jira/browse/HIVE-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-26651: Description: Same fix as > MultiDelimitSerDe shouldn't rely on default charset > --- > > Key: HIVE-26651 > URL: https://issues.apache.org/jira/browse/HIVE-26651 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > Same fix as -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26651) MultiDelimitSerDe shouldn't rely on default charset
[ https://issues.apache.org/jira/browse/HIVE-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-26651: Summary: MultiDelimitSerDe shouldn't rely on default charset (was: MultiDelimitSerDe should not rely on default charset) > MultiDelimitSerDe shouldn't rely on default charset > --- > > Key: HIVE-26651 > URL: https://issues.apache.org/jira/browse/HIVE-26651 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26651) MultiDelimitSerDe shouldn't rely on default charset
[ https://issues.apache.org/jira/browse/HIVE-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-26651: Description: Same fix as HIVE-26639 but on different codepath. (was: Same fix as ) > MultiDelimitSerDe shouldn't rely on default charset > --- > > Key: HIVE-26651 > URL: https://issues.apache.org/jira/browse/HIVE-26651 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > Same fix as HIVE-26639 but on different codepath. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26633) Make thrift max message size configurable
[
https://issues.apache.org/jira/browse/HIVE-26633?focusedWorklogId=818420=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818420
]
ASF GitHub Bot logged work on HIVE-26633:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 10:32
Start Date: 19/Oct/22 10:32
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3674:
URL: https://github.com/apache/hive/pull/3674#issuecomment-1283786693
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3674)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3674=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3674=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3674=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=CODE_SMELL)
[1 Code
Smell](https://sonarcloud.io/project/issues?id=apache_hive=3674=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3674=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3674=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 818420)
Time Spent: 4h (was: 3h 50m)
> Make thrift max message size configurable
> -
>
> Key: HIVE-26633
> URL: https://issues.apache.org/jira/browse/HIVE-26633
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Since thrift >= 0.14, thrift now enforces max message sizes through a
> TConfiguration object as described here:
> [https://github.com/apache/thrift/blob/master/doc/specs/thrift-tconfiguration.md]
> By default MaxMessageSize gets set to 100MB.
> As a result it is possible for HMS clients not to be able to retrieve certain
> metadata for tables with a large amount of partitions or other metadata.
> For example on a cluster configured with kerberos between hs2 and hms,
> querying a large table (10k partitions, 200 columns with names of 200
> characters) results in this backtrace:
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>
> at
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>
> at
[jira] [Work logged] (HIVE-26645) Preparing for 4.0.0 development
[ https://issues.apache.org/jira/browse/HIVE-26645?focusedWorklogId=818401=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818401 ] ASF GitHub Bot logged work on HIVE-26645: - Author: ASF GitHub Bot Created on: 19/Oct/22 10:00 Start Date: 19/Oct/22 10:00 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3685: URL: https://github.com/apache/hive/pull/3685#issuecomment-1283743147 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3685) [](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3685=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3685=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3685=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3685=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3685=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3685=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 818401) Time Spent: 1h (was: 50m) > Preparing for 4.0.0 development > --- > > Key: HIVE-26645 > URL: https://issues.apache.org/jira/browse/HIVE-26645 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=818396=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818396
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 09:40
Start Date: 19/Oct/22 09:40
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3656:
URL: https://github.com/apache/hive/pull/3656#discussion_r999198562
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnErrorMsg.java:
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.txn;
+
+public enum TxnErrorMsg {
+// Refer ErrorMsg.java
+// 5 to 5: Txn Errors Codes.
+NONE(5, " None "),
+ABORT_QUERY(50001, " Txn aborted by Abort Query Command "),
+ABORT_TIMEOUT(50002, " Txn aborted due to heartbeat time-out after "),
+ABORT_INTERRUPT(50003, " Txn aborted due to Interrupt Signal or user
aborted the query "),
+ABORT_REPL_TXN(50004, " Replication:" +
+"Abort all the allocated txns so that the mapped write ids are
referred as aborted ones. ");
+
+private final int errorCode;
+private final String txnErrorMsg;
+
+TxnErrorMsg(int errorCode, String txnErrorMsg) {
+this.errorCode = errorCode;
+this.txnErrorMsg = txnErrorMsg;
+}
+
+public int getErrorCode() {
+return errorCode;
+}
+
+public String getTxnErrorMsg() {
Review Comment:
is it used at all? how would it work? see
static private HiveException getHiveException(Exception e, String msg,
String logMsg) {
ErrorMsg errorMsg = ErrorMsg.getErrorMsg(e);
}
Issue Time Tracking
---
Worklog Id: (was: 818396)
Time Spent: 1h 50m (was: 1h 40m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Should add TXNS.COMMENT field or something like that so that if the system
> aborts a transaction (due to timeout, for example) we can add a message to
> that effect to the aborted transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=818390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818390
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 09:38
Start Date: 19/Oct/22 09:38
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3656:
URL: https://github.com/apache/hive/pull/3656#discussion_r999193525
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -1023,6 +1023,7 @@ public void abortTxn(AbortTxnRequest rqst) throws
NoSuchTxnException, MetaExcept
long sourceTxnId = -1;
boolean isReplayedReplTxn =
TxnType.REPL_CREATED.equals(rqst.getTxn_type());
boolean isHiveReplTxn = rqst.isSetReplPolicy() &&
TxnType.DEFAULT.equals(rqst.getTxn_type());
+String abrotMsg = (!rqst.isSetErrorMessage() && rqst.isSetTxnid())?
TxnErrorMsg.ABORT_QUERY.name():rqst.getErrorMessage();
Review Comment:
why do we need to check for the txnId here? In thrift, txnid is set as
mandatory
struct AbortTxnRequest {
1: required i64 txnid,
}
Issue Time Tracking
---
Worklog Id: (was: 818390)
Time Spent: 1h 40m (was: 1.5h)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Should add TXNS.COMMENT field or something like that so that if the system
> aborts a transaction (due to timeout, for example) we can add a message to
> that effect to the aborted transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[ https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=818389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818389 ] ASF GitHub Bot logged work on HIVE-11495: - Author: ASF GitHub Bot Created on: 19/Oct/22 09:36 Start Date: 19/Oct/22 09:36 Worklog Time Spent: 10m Work Description: deniskuzZ commented on code in PR #3656: URL: https://github.com/apache/hive/pull/3656#discussion_r999193525 ## standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java: ## @@ -1023,6 +1023,7 @@ public void abortTxn(AbortTxnRequest rqst) throws NoSuchTxnException, MetaExcept long sourceTxnId = -1; boolean isReplayedReplTxn = TxnType.REPL_CREATED.equals(rqst.getTxn_type()); boolean isHiveReplTxn = rqst.isSetReplPolicy() && TxnType.DEFAULT.equals(rqst.getTxn_type()); +String abrotMsg = (!rqst.isSetErrorMessage() && rqst.isSetTxnid())? TxnErrorMsg.ABORT_QUERY.name():rqst.getErrorMessage(); Review Comment: why do we need to check for the txnId here? Issue Time Tracking --- Worklog Id: (was: 818389) Time Spent: 1.5h (was: 1h 20m) > Add aborted reason to transaction information. > -- > > Key: HIVE-11495 > URL: https://issues.apache.org/jira/browse/HIVE-11495 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Should add TXNS.COMMENT field or something like that so that if the system > aborts a transaction (due to timeout, for example) we can add a message to > that effect to the aborted transaction. > Another reason: Commit can fail due to a conflicting write from another txn > (since HIVE-13395) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26210) Fix tests for Cleaner failed attempt threshold
[ https://issues.apache.org/jira/browse/HIVE-26210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh updated HIVE-26210: --- Fix Version/s: 4.0.0-alpha-2 > Fix tests for Cleaner failed attempt threshold > -- > > Key: HIVE-26210 > URL: https://issues.apache.org/jira/browse/HIVE-26210 > Project: Hive > Issue Type: Bug >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26352) Tez queue access check fails with GSS Exception on Compaction
[ https://issues.apache.org/jira/browse/HIVE-26352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh updated HIVE-26352: --- Fix Version/s: 4.0.0-alpha-2 > Tez queue access check fails with GSS Exception on Compaction > - > > Key: HIVE-26352 > URL: https://issues.apache.org/jira/browse/HIVE-26352 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 1h > Remaining Estimate: 0h > > When hive.server2.tez.queue.access.check is enabled and > hive.compactor.job.queue is configured to a different queue than default, > compaction jobs fails at analyze table step, but it's marked as compacted. It > seems that the application is trying to call the yarn queue check url with > wrong credentials -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=818388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818388
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 09:34
Start Date: 19/Oct/22 09:34
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3656:
URL: https://github.com/apache/hive/pull/3656#discussion_r999192218
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##
@@ -8699,6 +8700,7 @@ public OpenTxnsResponse open_txns(OpenTxnRequest rqst)
throws TException {
@Override
public void abort_txn(AbortTxnRequest rqst) throws TException {
+rqst.setErrorMessage("ABORT_INTERRUPT");
Review Comment:
can't we use TxnErrorMsg.ABORT_INTERRUPT.name() instead of hardcoded literal?
Issue Time Tracking
---
Worklog Id: (was: 818388)
Time Spent: 1h 20m (was: 1h 10m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Should add TXNS.COMMENT field or something like that so that if the system
> aborts a transaction (due to timeout, for example) we can add a message to
> that effect to the aborted transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-25993) Query-based compaction doesn't work when partition column type is boolean
[ https://issues.apache.org/jira/browse/HIVE-25993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh updated HIVE-25993: --- Fix Version/s: 4.0.0-alpha-2 > Query-based compaction doesn't work when partition column type is boolean > - > > Key: HIVE-25993 > URL: https://issues.apache.org/jira/browse/HIVE-25993 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 20m > Remaining Estimate: 0h > > Query based compaction fails on tables with boolean partition column. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26026) Use the new "REFUSED" compaction state where it makes sense
[ https://issues.apache.org/jira/browse/HIVE-26026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh updated HIVE-26026: --- Fix Version/s: 4.0.0-alpha-2 > Use the new "REFUSED" compaction state where it makes sense > --- > > Key: HIVE-26026 > URL: https://issues.apache.org/jira/browse/HIVE-26026 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 6.5h > Remaining Estimate: 0h > > The > org.apache.hadoop.hive.ql.txn.compactor.Worker#findNextCompactionAndExecute > method does several checks (The table/partition exists, is not sorted, there > are enough files to compact, etc.) before it actually executes the compaction > request. If the compaction request fails on any of these checks, it is put to > "SUCCEEDED" state which is often misleading for users. SHOW COMPACTIONS will > show these requests as succeeded without an error, while the table is not > compacted at all. > For these cases, the state should be "REFUSED" instead of "SUCCEEDED" among > with the appropriate error message. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26059) Eventually clean compactions in "refused" state from compaction history
[ https://issues.apache.org/jira/browse/HIVE-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh updated HIVE-26059: --- Fix Version/s: 4.0.0-alpha-2 > Eventually clean compactions in "refused" state from compaction history > --- > > Key: HIVE-26059 > URL: https://issues.apache.org/jira/browse/HIVE-26059 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Compactions in states succeeded, failed, and did not initiate have a > retention threshold (example: > metastore.compactor.history.retention.succeeded) and are purged from > COMPLETED_COMPACTIONS if the number of compactions in this state per > partition/unpartitioned table passes the threshold. This keeps the size of > COMPLETED_COMPACTIONS in check. > We should also purge refused compactions from COMPLETED_COMPACTIONS. > See: > CompactionTxnHandler#purgeCompactionHistory > ! Also: REFUSED_RESPONSE should be added to > org.apache.hadoop.hive.metastore.txn.TxnStore#COMPACTION_STATES so that > metrics will be collected about it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=818385=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818385
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 19/Oct/22 09:33
Start Date: 19/Oct/22 09:33
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3656:
URL: https://github.com/apache/hive/pull/3656#discussion_r999190297
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -5085,16 +5087,21 @@ private int abortTxns(Connection dbConn, List
txnids, boolean checkHeartbe
StringBuilder prefix = new StringBuilder();
StringBuilder suffix = new StringBuilder();
- // add update txns queries to query list
- prefix.append("UPDATE \"TXNS\" SET \"TXN_STATE\" =
").append(TxnStatus.ABORTED)
- .append(" WHERE \"TXN_STATE\" =
").append(TxnStatus.OPEN).append(" AND ");
if (checkHeartbeat) {
-suffix.append(" AND \"TXN_LAST_HEARTBEAT\" < ")
-.append(getEpochFn(dbProduct)).append("-").append(timeout);
+LOG.info(TxnErrorMsg.ABORT_TIMEOUT.toString() + timeout);
Review Comment:
can we keep the generic part and just add extra stuff under the if section?
Issue Time Tracking
---
Worklog Id: (was: 818385)
Time Spent: 1h 10m (was: 1h)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Should add TXNS.COMMENT field or something like that so that if the system
> aborts a transaction (due to timeout, for example) we can add a message to
> that effect to the aborted transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26650) Run qtests with different default encoding
[
https://issues.apache.org/jira/browse/HIVE-26650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620120#comment-17620120
]
Stamatis Zampetakis commented on HIVE-26650:
Maybe instead of changing globally the default encoding with surefire we could
configure it per-test by introducing a new qt option in a similar way that we
do for the default timezone (introduced by HIVE-24074).
{noformat}
--! qt:timezone:Asia/Singapore
{noformat}
> Run qtests with different default encoding
> --
>
> Key: HIVE-26650
> URL: https://issues.apache.org/jira/browse/HIVE-26650
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Priority: Major
>
> {code}
> org.apache.maven.plugins
> maven-surefire-plugin
>
> + -Dfile.encoding=US-ASCII
> {code}
> and we expect the qtests (with special UTF8 chars) to produce the very same
> output ==> ensure that character encodings aren't messed up
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22888:
--
Fix Version/s: 4.0.0-alpha-1
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Fix For: 4.0.0-alpha-1
>
> Attachments: HIVE-22888.1.patch, HIVE-22888.10.patch,
> HIVE-22888.11.patch, HIVE-22888.2.patch, HIVE-22888.3.patch,
> HIVE-22888.4.patch, HIVE-22888.5.patch, HIVE-22888.6.patch,
> HIVE-22888.8.patch, HIVE-22888.9.patch, acid-lock-perf-test.pdf
>
>
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE,
> HL_LOCK_TYPE FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 333) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE
> HL_LOCK_EXT_ID = 333) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LBC.HL_TABLE IS NULL OR LS.HL_TABLE =
> LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NULL OR
> LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE (LBC.HL_TXNID = 0 OR LS.HL_TXNID != LBC.HL_TXNID)
> AND (LBC.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NULL AND LS.HL_LOCK_TYPE='r' AND LBC.HL_TABLE
> IS NOT NULL )
> OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
> OR LBC.HL_LOCK_TYPE='r' AND LS.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NOT NULL AND LBC.HL_TABLE IS NULL))
> LIMIT 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-23503) ValidTxnManager doesn't consider txns opened and committed between snapshot generation and locking when evaluating ValidTxnListState
[ https://issues.apache.org/jira/browse/HIVE-23503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23503: -- Fix Version/s: 4.0.0-alpha-1 > ValidTxnManager doesn't consider txns opened and committed between snapshot > generation and locking when evaluating ValidTxnListState > > > Key: HIVE-23503 > URL: https://issues.apache.org/jira/browse/HIVE-23503 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Fix For: 4.0.0-alpha-1 > > Attachments: HIVE-23503.1.patch > > > ValidTxnManager doesn't consider txns opened and committed between snapshot > generation and locking when evaluating ValidTxnListState. This cause issues > like duplicate insert in case of concurrent merge insert & insert. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23349:
--
Fix Version/s: 4.0.0-alpha-1
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Fix For: 4.0.0-alpha-1
>
> Attachments: HIVE-23349.1.patch, HIVE-23349.2.patch,
> HIVE-23349.3.patch, HIVE-23349.4.patch, HIVE-23349.5.patch, HIVE-23349.6.patch
>
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or
> EXCL_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-23832) Compaction cleaner fails to clean up deltas when using blocking compaction
[
https://issues.apache.org/jira/browse/HIVE-23832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23832:
--
Fix Version/s: 4.0.0-alpha-1
> Compaction cleaner fails to clean up deltas when using blocking compaction
> --
>
> Key: HIVE-23832
> URL: https://issues.apache.org/jira/browse/HIVE-23832
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> {code}
> CREATE TABLE default.compcleanup (
>cda_id int,
>cda_run_id varchar(255),
>cda_load_tstimestamp,
>global_party_idstring,
>group_id string)
> COMMENT 'gp_2_gr'
> PARTITIONED BY (
>cda_date int,
>cda_job_name varchar(12))
> STORED AS ORC;
> -- cda_date=20200601/cda_job_name=core_base
> INSERT INTO default.compcleanup VALUES
> (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
> SELECT * FROM default.compcleanup where cda_date = 20200601 and cda_job_name
> = 'core_base';
> ALTER TABLE default.compcleanup PARTITION (cda_date=20200601,
> cda_job_name='core_base') COMPACT 'MAJOR' AND WAIT;
> {code}
> When using blocking compaction Cleaner skips processing due to the presence
> of open txn (by `ALTER TABLE`) below Compactor's one.
> {code}
> AcidUtils - getChildState() ignoring([])
> pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_002_v035
> {code}
> AcidUtils.processBaseDir
> {code}
> if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted,
> validTxnList)) {
>return;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23032: -- Fix Version/s: 4.0.0-alpha-1 > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Fix For: 4.0.0-alpha-1 > > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch, HIVE-23032.8.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-23293) Locks: Implement zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23293: -- Fix Version/s: 4.0.0-alpha-1 > Locks: Implement zero-wait readers > --- > > Key: HIVE-23293 > URL: https://issues.apache.org/jira/browse/HIVE-23293 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Fix For: 4.0.0-alpha-1 > > Attachments: HIVE-23293.1.patch, HIVE-23293.2.patch, > HIVE-23293.3.patch > > > With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not > have to wait for any lock - it can fails fast for a pending EXCLUSIVE, > because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no > semantic reason to wait for them. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-24125) Incorrect transaction snapshot invalidation with unnecessary writeset check for exclusive operations
[ https://issues.apache.org/jira/browse/HIVE-24125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-24125: -- Fix Version/s: 4.0.0-alpha-1 > Incorrect transaction snapshot invalidation with unnecessary writeset check > for exclusive operations > > > Key: HIVE-24125 > URL: https://issues.apache.org/jira/browse/HIVE-24125 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Fix For: 4.0.0-alpha-1 > > > Fixes [HIVE-23725|https://issues.apache.org/jira/browse/HIVE-23725] and > addresses issue with concurrent exclusive writes (shouldn't fail on writeset > check). > https://docs.google.com/document/d/1NVfk479_SxVIWPLXYmZkU8MYQE5nhcHbKMrf3bO_qwI -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-22906) Redundant checkLock Mutex blocks concurrent Lock requests
[ https://issues.apache.org/jira/browse/HIVE-22906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22906: -- Fix Version/s: 4.0.0-alpha-1 > Redundant checkLock Mutex blocks concurrent Lock requests > - > > Key: HIVE-22906 > URL: https://issues.apache.org/jira/browse/HIVE-22906 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Fix For: 4.0.0-alpha-1 > > Attachments: HIVE-22906.1.patch > > > enqueueLocks is already serialising locks creation via (SELECT NL_NEXT FROM > NEXT_LOCK_ID FOR UPDATE). Requested locks would be assigned 'W' state. > checkLock is iterating over the sorted set of conflicting locks below current > EXT_LOCK_ID. It does handle the situation when there is conflicting lock with > lower ID in 'W' state - lock request would be denied and retried later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-24000) Put exclusive MERGE INSERT under the feature flag
[ https://issues.apache.org/jira/browse/HIVE-24000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-24000: -- Fix Version/s: 4.0.0-alpha-1 > Put exclusive MERGE INSERT under the feature flag > - > > Key: HIVE-24000 > URL: https://issues.apache.org/jira/browse/HIVE-24000 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-24374) Do not remove metadata for aborted writes that were batched with valid ones in the same delta file
[ https://issues.apache.org/jira/browse/HIVE-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-24374: -- Fix Version/s: 4.0.0-alpha-1 > Do not remove metadata for aborted writes that were batched with valid ones > in the same delta file > -- > > Key: HIVE-24374 > URL: https://issues.apache.org/jira/browse/HIVE-24374 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Fix For: 4.0.0-alpha-1 > > > BUGFIX for streaming use-case -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-24667) Truncate optimization to avoid unnecessary per partition DB get operations
[ https://issues.apache.org/jira/browse/HIVE-24667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-24667: -- Fix Version/s: 4.0.0-alpha-1 > Truncate optimization to avoid unnecessary per partition DB get operations > -- > > Key: HIVE-24667 > URL: https://issues.apache.org/jira/browse/HIVE-24667 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-24781) Allow to use custom queue for query based compaction
[ https://issues.apache.org/jira/browse/HIVE-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-24781: -- Fix Version/s: 4.0.0-alpha-1 > Allow to use custom queue for query based compaction > > > Key: HIVE-24781 > URL: https://issues.apache.org/jira/browse/HIVE-24781 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 40m > Remaining Estimate: 0h > > Provide a change such that all compaction jobs, both fully acid and insert > only are submitted to a dedicated queue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-24211) Replace Snapshot invalidate logic with WriteSet check for txn conflict detection
[ https://issues.apache.org/jira/browse/HIVE-24211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-24211: -- Fix Version/s: 4.0.0-alpha-1 > Replace Snapshot invalidate logic with WriteSet check for txn conflict > detection > > > Key: HIVE-24211 > URL: https://issues.apache.org/jira/browse/HIVE-24211 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 50m > Remaining Estimate: 0h > > *Issue with concurrent writes on partitioned table:* > Concurrent writes on different partitions should execute in parallel without > issues. They acquire a shared lock on table level and exclusive write on > partition level (hive.txn.xlock.write=true). > However there is a problem with the Snapshot validation. It compares valid > writeIds seen by current transaction, recorded before locking, with the > actual list of writeIds. The Issue is that writeId in Snapshot has no > information on partition, meaning that concurrent writes to different > partitions would be seen as writes to the same non-partitioned table causing > Snapshot to be obsolete. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-24906) Suffix the table location with UUID/txnId
[ https://issues.apache.org/jira/browse/HIVE-24906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-24906: -- Fix Version/s: 4.0.0-alpha-1 > Suffix the table location with UUID/txnId > - > > Key: HIVE-24906 > URL: https://issues.apache.org/jira/browse/HIVE-24906 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Suffixing the table location during create table with UUID/txnId can help in > deleting the data in asynchronous fashion. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25502) Cleaner causes data loss when processing aborted txn with DP
[ https://issues.apache.org/jira/browse/HIVE-25502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25502: -- Fix Version/s: 4.0.0-alpha-1 > Cleaner causes data loss when processing aborted txn with DP > - > > Key: HIVE-25502 > URL: https://issues.apache.org/jira/browse/HIVE-25502 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 1h > Remaining Estimate: 0h > > Steps to reproduce: > 1) CREATE TABLE cws (b STRING) PARTITIONED BY (a INT) STORED AS ORC; > 2) insert into cws partition (a) values (1, '1'); > update csw set b='2' where a=1; > 3) open HiveStreamingConnection with batch size set to 1, begin txn, write > something and abort the txn > 4) insert into cws partition (a) values (2, '2'); > update csw set b='3' where a=2; > 5) manually trigger the compaction and verify that data for part a=1 is gone -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25217) Move isEligibleForCompaction evaluation under the Initiator thread pool
[ https://issues.apache.org/jira/browse/HIVE-25217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25217: -- Fix Version/s: 4.0.0-alpha-1 > Move isEligibleForCompaction evaluation under the Initiator thread pool > --- > > Key: HIVE-25217 > URL: https://issues.apache.org/jira/browse/HIVE-25217 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 20m > Remaining Estimate: 0h > > Checking for eligibility >1 mil of distinct table / partition combinations > can take a while by the Initiator since all steps are performed in the main > thread. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25503) Add cleanup for the duplicate COMPLETED_TXN_COMPONENTS entries
[ https://issues.apache.org/jira/browse/HIVE-25503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25503: -- Fix Version/s: 4.0.0-alpha-1 > Add cleanup for the duplicate COMPLETED_TXN_COMPONENTS entries > -- > > Key: HIVE-25503 > URL: https://issues.apache.org/jira/browse/HIVE-25503 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Performace improvement. Accumulated entries in COMPLETED_TXN_COMPONENTS can > lead to query performance degradation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25441) Incorrect deltas split for sub-compactions when using `hive.compactor.max.num.delta`
[
https://issues.apache.org/jira/browse/HIVE-25441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-25441:
--
Fix Version/s: 4.0.0-alpha-1
> Incorrect deltas split for sub-compactions when using
> `hive.compactor.max.num.delta`
>
>
> Key: HIVE-25441
> URL: https://issues.apache.org/jira/browse/HIVE-25441
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {code}
> #Repro steps:
> #1./ set hive.compactor.max.num.delta to 5 on HMS
> #2./ Set up the table
> set hive.merge.cardinality.check=false;
> create table test (k int);
> ALTER TABLE test SET TBLPROPERTIES ('NO_AUTO_COMPACTION'='true');
> insert into test values (1);
> alter table test compact 'major' and wait;
> dfs -ls '/warehouse/tablespace/managed/hive/test';
> # drwxrwx---+ - hive hive 0 2021-08-09 12:26
> /warehouse/tablespace/managed/hive/test/base_008_v416
> select * from test;
> # k=1
> #run 3 times so there's enough delta dirs, ie. 6 (should just increase k by 1)
> #basically just removes the row and adds a new row with k+1 value
> MERGE INTO test AS T USING (select * from test union all select k+1 from
> test) AS S
> ON T.k=s.k
> WHEN MATCHED THEN DELETE
> WHEN not MATCHED THEN INSERT values (s.k);
> select * from test;
> #k=4
> dfs -ls '/warehouse/tablespace/managed/hive/test';
> #drwxrwx---+ - hive hive 0 2021-08-09 12:26
> /warehouse/tablespace/managed/hive/test/base_008_v416
> #drwxrwx---+ - hive hive 0 2021-08-09 12:28
> /warehouse/tablespace/managed/hive/test/delete_delta_009_009_0001
> #drwxrwx---+ - hive hive 0 2021-08-09 12:29
> /warehouse/tablespace/managed/hive/test/delete_delta_010_010_0001
> #drwxrwx---+ - hive hive 0 2021-08-09 12:29
> /warehouse/tablespace/managed/hive/test/delete_delta_011_011_0001
> #drwxrwx---+ - hive hive 0 2021-08-09 12:28
> /warehouse/tablespace/managed/hive/test/delta_009_009_0003
> #drwxrwx---+ - hive hive 0 2021-08-09 12:29
> /warehouse/tablespace/managed/hive/test/delta_010_010_0003
> #drwxrwx---+ - hive hive 0 2021-08-09 12:29
> /warehouse/tablespace/managed/hive/test/delta_011_011_0003
> alter table test compact 'major' and wait;
> select * from test;
> #result is empty
> dfs -ls '/warehouse/tablespace/managed/hive/test';
> #2drwxrwx---+ - hive hive 0 2021-08-09 12:31
> /warehouse/tablespace/managed/hive/test/base_011_v428
> {code}
> Some logs from the above example:
> {code}
> 2021-08-09 12:30:37,532 WARN
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR:
> [nightly-7x-us-2-2.nightly-7x-us-2.root.hwx.site-49_executor]: 6 delta files
> found for default.test located at
> hdfs://nightly-7x-us-2-2.nightly-7x-us-2.root.hwx.site:8020/warehouse/tablespace/managed/hive/test!
> This is likely a sign of misconfiguration, especially if this message
> repeats. Check that compaction is running properly. Check for any
> runaway/mis-configured process writing to ACID tables, especially using
> Streaming Ingest API.
> 2021-08-09 12:30:37,533 INFO
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR:
> [nightly-7x-us-2-2.nightly-7x-us-2.root.hwx.site-49_executor]: Submitting
> MINOR compaction job
> 'nightly-7x-us-2-2.nightly-7x-us-2.root.hwx.site-49-compactor-default.test_0'
> to default queue. (current delta dirs count=5, obsolete delta dirs count=-1.
> TxnIdRange[9,11]
> 2021-08-09 12:30:38,003 INFO
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR:
> [nightly-7x-us-2-2.nightly-7x-us-2.root.hwx.site-49_executor]: Submitted
> compaction job
> 'nightly-7x-us-2-2.nightly-7x-us-2.root.hwx.site-49-compactor-default.test_0'
> with jobID=job_1628497133224_0051 compaction ID=23
> #From app logs of the minor compaction, note that delta_011_011_0001
> is missing from the list
> 2021-08-09 12:30:47,399 INFO [main] org.apache.hadoop.mapred.MapTask:
> Processing split: CompactorInputSplit{base: null, bucket: 0, length: 3231,
> deltas: [delete_delta_009_009_0001, delta_009_009_0003,
> delete_delta_010_010_0001, delta_010_010_0003,
> delete_delta_011_011_0001]}
> 2021-08-09 12:30:53,061 INFO
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR:
> [nightly-7x-us-2-2.nightly-7x-us-2.root.hwx.site-49_executor]: Submitting
> MAJOR compaction job
> 'nightly-7x-us-2-2.nightly-7x-us-2.root.hwx.site-49-compactor-default.test'
> to default queue. (current delta dirs count=2, obsolete delta dirs count=6.
>
[jira] [Updated] (HIVE-25393) Prevent concurrent compaction and DDL operations
[ https://issues.apache.org/jira/browse/HIVE-25393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25393: -- Fix Version/s: 4.0.0-alpha-1 > Prevent concurrent compaction and DDL operations > > > Key: HIVE-25393 > URL: https://issues.apache.org/jira/browse/HIVE-25393 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 50m > Remaining Estimate: 0h > > Introduce read locks in compaction (MR, Minor Query-based) that would prevent > concurrency issues with DDL operations (DROP/RENAME table). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25710) Config used to enable non-blocking TRUNCATE is not properly propagated
[ https://issues.apache.org/jira/browse/HIVE-25710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25710: -- Fix Version/s: 4.0.0-alpha-1 > Config used to enable non-blocking TRUNCATE is not properly propagated > -- > > Key: HIVE-25710 > URL: https://issues.apache.org/jira/browse/HIVE-25710 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25781) Restore multi-threaded support in Cleaner after HIVE-25115
[ https://issues.apache.org/jira/browse/HIVE-25781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25781: -- Fix Version/s: 4.0.0-alpha-1 > Restore multi-threaded support in Cleaner after HIVE-25115 > -- > > Key: HIVE-25781 > URL: https://issues.apache.org/jira/browse/HIVE-25781 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 10m > Remaining Estimate: 0h > > In the case of accumulated entries for the same table/partition, they should > be processed one by one in ASC order of write_id's. > However, to support multi-threaded processing in the Cleaner, we need to make > sure that the entries from the same group are not processed concurrently by > multiple threads. > One of the solutions that don't involve synchronization (check if already in > flight, re-try when possible) would be to move the entries from the same > group to the next Cleaner cycle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25688) Non blocking DROP PARTITION implementation
[ https://issues.apache.org/jira/browse/HIVE-25688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25688: -- Fix Version/s: 4.0.0-alpha-1 > Non blocking DROP PARTITION implementation > -- > > Key: HIVE-25688 > URL: https://issues.apache.org/jira/browse/HIVE-25688 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > This changes the drop partition operation to asynchronous. The data files of > transactional tables will not be deleted, but a new truncated basefile will > be written, which is going to be later cleaned up by the Compactor/Cleaner. > This along with a few other changes will enable us to not use read locks, > which provides perf boost to the transactional tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25881) isEligibleForCompaction should be executed under the custom threadPool
[ https://issues.apache.org/jira/browse/HIVE-25881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25881: -- Fix Version/s: 4.0.0-alpha-1 > isEligibleForCompaction should be executed under the custom threadPool > -- > > Key: HIVE-25881 > URL: https://issues.apache.org/jira/browse/HIVE-25881 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25631) Initiator speed-up: only read compaction history once per loop
[ https://issues.apache.org/jira/browse/HIVE-25631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25631: -- Fix Version/s: 4.0.0-alpha-1 > Initiator speed-up: only read compaction history once per loop > -- > > Key: HIVE-25631 > URL: https://issues.apache.org/jira/browse/HIVE-25631 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 50m > Remaining Estimate: 0h > > In checkFailedCompactions (which is called for every partition in the list of > potentials to compact) we select from metadata table COMPLETED_COMPACTIONS. > But the Initiator main loop already has a ShowCompactResponse. We can use > that instead. > For cases where metadata is huge, this will help speed up the Initiator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-25904) ObjectStore's updateTableColumnStatistics is not ThreadSafe
[
https://issues.apache.org/jira/browse/HIVE-25904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko reassigned HIVE-25904:
-
Assignee: Denys Kuzmenko
> ObjectStore's updateTableColumnStatistics is not ThreadSafe
> ---
>
> Key: HIVE-25904
> URL: https://issues.apache.org/jira/browse/HIVE-25904
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> {code}
> [root@igansperger-hive-tgt-3 ~]# cat test.sh
> hive -e 'create database test; create external table test.foo(col1 string);'
> 2> /dev/null
> hive -e "select count(*) from sys.tab_col_stats where db_name = 'test' and
> table_name = 'foo'" 2> /dev/null
> export JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
> export JAVA_OPTS="-Xmx1g"
> export PATH="/root/scala-2.13.8/bin:$JAVA_HOME/bin:$PATH"
> export CONF_DIR=/run/cloudera-scm-agent/process/79-hive_on_tez-HIVESERVER2
> export CDH_HCAT_HOME=/opt/cloudera/parcels/CDH/lib/hive-hcatalog/
> export CDH_HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
> CLASSPATH="$CLASSPATH:$CONF_DIR/hadoop-conf"
> CLASSPATH="$CLASSPATH:$CONF_DIR/hive-conf"
> CLASSPATH="$CLASSPATH:$(hadoop classpath)"
> CLASSPATH="$CLASSPATH:$CDH_HIVE_HOME/*"
> CLASSPATH="$CLASSPATH:$CDH_HIVE_HOME/lib/*"
> CLASSPATH="$CLASSPATH:${CDH_HCAT_HOME}/share/webhcat/java-client/hive-webhcat-java-client.jar"
> CLASSPATH="$CLASSPATH:${CDH_HCAT_HOME}/share/hcatalog/hive-hcatalog-core.jar"
> scala -classpath $CLASSPATH <<-EOF
> import org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> import org.apache.hadoop.hive.conf.HiveConf
> import org.apache.hadoop.hive.metastore.api._
> def go() = {
> val conf = new HiveConf()
> val client = new HiveMetaStoreClient(conf)
> val colStatData = new ColumnStatisticsData()
> colStatData.setStringStats(new StringColumnStatsData(3, 3.0, 0, 1))
> val colStatsObj = new ColumnStatisticsObj("col1", "string", colStatData)
> val colStatsObjs = java.util.Arrays.asList(colStatsObj)
> val colStatsDesc = new ColumnStatisticsDesc(true, "test", "foo")
> val colStats = new ColumnStatistics(colStatsDesc, colStatsObjs)
> colStats.setEngine("hive")
> client.updateTableColumnStatistics(colStats)
> println("SUCCESS")
> }
> val t1 = new Thread(() => go())
> val t2 = new Thread(() => go())
> t1.start()
> t2.start()
> t1.join()
> t2.join()
> go()
> EOF
> hive -e "select count(*) from sys.tab_col_stats where db_name = 'test' and
> table_name = 'foo'" 2> /dev/null
> {code}
> This produces (minus logging):
> {code}
> [root@igansperger-hive-tgt-3 ~]# sh test.sh
> +--+
> | _c0 |
> +--+
> | 0|
> +--+
> Welcome to Scala 2.13.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_232).
> Type in expressions for evaluation. Or try :help.
> SUCCESS
> SUCCESS
> org.apache.hadoop.hive.metastore.api.MetaException: Unexpected 2 statistics
> for 1 columns
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$update_table_column_statistics_req_result$update_table_column_statistics_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$update_table_column_statistics_req_result$update_table_column_statistics_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$update_table_column_statistics_req_result.read(ThriftHiveMetastore.java)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_update_table_column_statistics_req(ThriftHiveMetastore.java:4597)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.update_table_column_statistics_req(ThriftHiveMetastore.java:4584)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.updateTableColumnStatistics(HiveMetaStoreClient.java:2846)
> at go(:13)
> ... 32 elided
> scala>
> scala> :quit
> +--+
> | _c0 |
> +--+
> | 2|
> +--+
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-25934) Non blocking RENAME PARTITION implementation
[ https://issues.apache.org/jira/browse/HIVE-25934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25934: -- Fix Version/s: 4.0.0-alpha-1 > Non blocking RENAME PARTITION implementation > > > Key: HIVE-25934 > URL: https://issues.apache.org/jira/browse/HIVE-25934 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Implement RENAME PARTITION in a way that doesn't have to wait for currently > running read operations to be finished. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25904) ObjectStore's updateTableColumnStatistics is not ThreadSafe
[
https://issues.apache.org/jira/browse/HIVE-25904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-25904:
--
Fix Version/s: 4.0.0-alpha-1
> ObjectStore's updateTableColumnStatistics is not ThreadSafe
> ---
>
> Key: HIVE-25904
> URL: https://issues.apache.org/jira/browse/HIVE-25904
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> {code}
> [root@igansperger-hive-tgt-3 ~]# cat test.sh
> hive -e 'create database test; create external table test.foo(col1 string);'
> 2> /dev/null
> hive -e "select count(*) from sys.tab_col_stats where db_name = 'test' and
> table_name = 'foo'" 2> /dev/null
> export JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
> export JAVA_OPTS="-Xmx1g"
> export PATH="/root/scala-2.13.8/bin:$JAVA_HOME/bin:$PATH"
> export CONF_DIR=/run/cloudera-scm-agent/process/79-hive_on_tez-HIVESERVER2
> export CDH_HCAT_HOME=/opt/cloudera/parcels/CDH/lib/hive-hcatalog/
> export CDH_HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
> CLASSPATH="$CLASSPATH:$CONF_DIR/hadoop-conf"
> CLASSPATH="$CLASSPATH:$CONF_DIR/hive-conf"
> CLASSPATH="$CLASSPATH:$(hadoop classpath)"
> CLASSPATH="$CLASSPATH:$CDH_HIVE_HOME/*"
> CLASSPATH="$CLASSPATH:$CDH_HIVE_HOME/lib/*"
> CLASSPATH="$CLASSPATH:${CDH_HCAT_HOME}/share/webhcat/java-client/hive-webhcat-java-client.jar"
> CLASSPATH="$CLASSPATH:${CDH_HCAT_HOME}/share/hcatalog/hive-hcatalog-core.jar"
> scala -classpath $CLASSPATH <<-EOF
> import org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> import org.apache.hadoop.hive.conf.HiveConf
> import org.apache.hadoop.hive.metastore.api._
> def go() = {
> val conf = new HiveConf()
> val client = new HiveMetaStoreClient(conf)
> val colStatData = new ColumnStatisticsData()
> colStatData.setStringStats(new StringColumnStatsData(3, 3.0, 0, 1))
> val colStatsObj = new ColumnStatisticsObj("col1", "string", colStatData)
> val colStatsObjs = java.util.Arrays.asList(colStatsObj)
> val colStatsDesc = new ColumnStatisticsDesc(true, "test", "foo")
> val colStats = new ColumnStatistics(colStatsDesc, colStatsObjs)
> colStats.setEngine("hive")
> client.updateTableColumnStatistics(colStats)
> println("SUCCESS")
> }
> val t1 = new Thread(() => go())
> val t2 = new Thread(() => go())
> t1.start()
> t2.start()
> t1.join()
> t2.join()
> go()
> EOF
> hive -e "select count(*) from sys.tab_col_stats where db_name = 'test' and
> table_name = 'foo'" 2> /dev/null
> {code}
> This produces (minus logging):
> {code}
> [root@igansperger-hive-tgt-3 ~]# sh test.sh
> +--+
> | _c0 |
> +--+
> | 0|
> +--+
> Welcome to Scala 2.13.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_232).
> Type in expressions for evaluation. Or try :help.
> SUCCESS
> SUCCESS
> org.apache.hadoop.hive.metastore.api.MetaException: Unexpected 2 statistics
> for 1 columns
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$update_table_column_statistics_req_result$update_table_column_statistics_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$update_table_column_statistics_req_result$update_table_column_statistics_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$update_table_column_statistics_req_result.read(ThriftHiveMetastore.java)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_update_table_column_statistics_req(ThriftHiveMetastore.java:4597)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.update_table_column_statistics_req(ThriftHiveMetastore.java:4584)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.updateTableColumnStatistics(HiveMetaStoreClient.java:2846)
> at go(:13)
> ... 32 elided
> scala>
> scala> :quit
> +--+
> | _c0 |
> +--+
> | 2|
> +--+
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-25956) Non blocking RENAME TABLE implementation
[ https://issues.apache.org/jira/browse/HIVE-25956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25956: -- Fix Version/s: 4.0.0-alpha-1 > Non blocking RENAME TABLE implementation > > > Key: HIVE-25956 > URL: https://issues.apache.org/jira/browse/HIVE-25956 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Implement ALTER TABLE RENAME in a way that doesn't have to wait for currently > running read operations to be finished. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26023) Non blocking REPLACE, RENAME COLUMNS
[ https://issues.apache.org/jira/browse/HIVE-26023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26023: -- Fix Version/s: 4.0.0-alpha-2 > Non blocking REPLACE, RENAME COLUMNS > > > Key: HIVE-26023 > URL: https://issues.apache.org/jira/browse/HIVE-26023 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 1h > Remaining Estimate: 0h > > Modify the REPLACE/RENAME COLUMNS operation to not acquire EXCLUSIVE lock > limiting the system concurrency. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26051) Non blocking ALTER MATERIALIZED VIEW REWRITE
[ https://issues.apache.org/jira/browse/HIVE-26051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26051: -- Fix Version/s: 4.0.0-alpha-2 > Non blocking ALTER MATERIALIZED VIEW REWRITE > > > Key: HIVE-26051 > URL: https://issues.apache.org/jira/browse/HIVE-26051 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 10m > Remaining Estimate: 0h > > AlterMaterializedViewRewriteAnalyzer does not change the view content It just > controls whether the view can be used in automatic query plan rewrite. No > need for an EXCLUSIVE lock here. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26052) Non blocking DROP MATERIALIZED VIEW implementation
[ https://issues.apache.org/jira/browse/HIVE-26052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26052: -- Fix Version/s: 4.0.0-alpha-2 > Non blocking DROP MATERIALIZED VIEW implementation > -- > > Key: HIVE-26052 > URL: https://issues.apache.org/jira/browse/HIVE-26052 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26053) Non blocking DROP VIEW
[ https://issues.apache.org/jira/browse/HIVE-26053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26053: -- Fix Version/s: 4.0.0-alpha-2 > Non blocking DROP VIEW > -- > > Key: HIVE-26053 > URL: https://issues.apache.org/jira/browse/HIVE-26053 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 40m > Remaining Estimate: 0h > > DROP VIEW shouldn't acquire any locks -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26106) Non blocking ADD, DROP constraint
[ https://issues.apache.org/jira/browse/HIVE-26106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26106: -- Fix Version/s: 4.0.0-alpha-2 > Non blocking ADD, DROP constraint > - > > Key: HIVE-26106 > URL: https://issues.apache.org/jira/browse/HIVE-26106 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 20m > Remaining Estimate: 0h > > No need for EXCLUSIVE lock here. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26129) Non blocking DROP CONNECTOR
[ https://issues.apache.org/jira/browse/HIVE-26129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26129: -- Fix Version/s: 4.0.0-alpha-2 (was: 4.0.0-alpha-1) > Non blocking DROP CONNECTOR > --- > > Key: HIVE-26129 > URL: https://issues.apache.org/jira/browse/HIVE-26129 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 20m > Remaining Estimate: 0h > > Use a less restrictive lock for data connectors, they do not have any > dependencies on other tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26076) Non blocking ADD PARTITION if not exists
[ https://issues.apache.org/jira/browse/HIVE-26076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26076: -- Fix Version/s: 4.0.0-alpha-2 > Non blocking ADD PARTITION if not exists > > > Key: HIVE-26076 > URL: https://issues.apache.org/jira/browse/HIVE-26076 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 1h > Remaining Estimate: 0h > > Originally X (EXCLUSIVE) lock was used to prevent races between concurrent > add partition calls with IF NOT EXISTS, however, the same behavior could be > achieved with a less restrictive X-write (EXCL_WRITE) lock. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26149) Non blocking DROP DATABASE implementation
[ https://issues.apache.org/jira/browse/HIVE-26149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26149: -- Fix Version/s: 4.0.0-alpha-2 (was: 4.0.0-alpha-1) > Non blocking DROP DATABASE implementation > - > > Key: HIVE-26149 > URL: https://issues.apache.org/jira/browse/HIVE-26149 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
