[jira] [Updated] (HIVE-22626) Fix TestStatsReplicationScenariosACIDNoAutogather
[
https://issues.apache.org/jira/browse/HIVE-22626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-22626:
--
Labels: pull-request-available (was: )
> Fix TestStatsReplicationScenariosACIDNoAutogather
> -
>
> Key: HIVE-22626
> URL: https://issues.apache.org/jira/browse/HIVE-22626
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Zoltan Haindrich
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Attachments: qalogs.tgz
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> this test is running "alone" because but still; it sometimes runs more than
> 40m which results in a timeout
> a jira search reveals that was pretty common:
> https://issues.apache.org/jira/issues/?jql=text%20~%20%22TestStatsReplicationScenariosACIDNoAutogather%22%20order%20by%20updated%20desc
> from the hive logs:
> * it seems like after a few minutes this test starts there is an exception:
> {code}
> 2019-12-10T22:43:19,594 DEBUG [Finalizer] metastore.HiveMetaStoreClient:
> Unable to shutdown metastore client. Will try closing transport directly.
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Socket closed
> at
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
> ~[libthrift-0.9.3-1.jar:0.9.3-1]
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
> ~[libthrift-0.9.3-1.jar:0.9.3-1]
> at
> org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66)
> ~[libthrift-0.9.3-1.jar:0.9.3-1]
> at
> com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:436)
> ~[libfb303-0.9.3.jar:?]
> at
> com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:430)
> ~[libfb303-0.9.3.jar:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:776)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy62.close(Unknown Source) [?:?]
> at org.apache.hadoop.hive.ql.metadata.Hive.close(Hive.java:542)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.finalize(Hive.java:514)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at java.lang.System$2.invokeFinalize(System.java:1270) [?:1.8.0_102]
> at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
> [?:1.8.0_102]
> at java.lang.ref.Finalizer.access$100(Finalizer.java:34) [?:1.8.0_102]
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)
> [?:1.8.0_102]
> Caused by: java.net.SocketException: Socket closed
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116)
> ~[?:1.8.0_102]
> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
> ~[?:1.8.0_102]
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[?:1.8.0_102]
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[?:1.8.0_102]
> at
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
> ~[libthrift-0.9.3-1.jar:0.9.3-1]
> {code}
> * after that some NoSuchObjectExceptions follow
> * and then some replications seems to happen
> I don't fully understand this; I'll attach the logs...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22626) Fix TestStatsReplicationScenariosACIDNoAutogather
[
https://issues.apache.org/jira/browse/HIVE-22626?focusedWorklogId=619741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619741
]
ASF GitHub Bot logged work on HIVE-22626:
-
Author: ASF GitHub Bot
Created on: 07/Jul/21 04:13
Start Date: 07/Jul/21 04:13
Worklog Time Spent: 10m
Work Description: ayushtkn opened a new pull request #2452:
URL: https://github.com/apache/hive/pull/2452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 619741)
Remaining Estimate: 0h
Time Spent: 10m
> Fix TestStatsReplicationScenariosACIDNoAutogather
> -
>
> Key: HIVE-22626
> URL: https://issues.apache.org/jira/browse/HIVE-22626
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Zoltan Haindrich
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: qalogs.tgz
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> this test is running "alone" because but still; it sometimes runs more than
> 40m which results in a timeout
> a jira search reveals that was pretty common:
> https://issues.apache.org/jira/issues/?jql=text%20~%20%22TestStatsReplicationScenariosACIDNoAutogather%22%20order%20by%20updated%20desc
> from the hive logs:
> * it seems like after a few minutes this test starts there is an exception:
> {code}
> 2019-12-10T22:43:19,594 DEBUG [Finalizer] metastore.HiveMetaStoreClient:
> Unable to shutdown metastore client. Will try closing transport directly.
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Socket closed
> at
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
> ~[libthrift-0.9.3-1.jar:0.9.3-1]
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
> ~[libthrift-0.9.3-1.jar:0.9.3-1]
> at
> org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66)
> ~[libthrift-0.9.3-1.jar:0.9.3-1]
> at
> com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:436)
> ~[libfb303-0.9.3.jar:?]
> at
> com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:430)
> ~[libfb303-0.9.3.jar:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:776)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy62.close(Unknown Source) [?:?]
> at org.apache.hadoop.hive.ql.metadata.Hive.close(Hive.java:542)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.finalize(Hive.java:514)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at java.lang.System$2.invokeFinalize(System.java:1270) [?:1.8.0_102]
> at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
> [?:1.8.0_102]
> at java.lang.ref.Finalizer.access$100(Finalizer.java:34) [?:1.8.0_102]
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)
> [?:1.8.0_102]
> Caused by: java.net.SocketException: Socket closed
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116)
> ~[?:1.8.0_102]
> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
> ~[?:1.8.0_102]
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[?:1.8.0_102]
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[?:1.8.0_102]
> at
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
> ~[libthrift-0.9.3-1.jar:0.9.3-1]
> {code}
> * after that some NoSuchObjectExceptions follow
> * and then some replications seems to happen
> I don't fully understand this; I'll attach the logs...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25307) Hive Server 2 crashes when Thrift library encounters particular security protocol issue
[ https://issues.apache.org/jira/browse/HIVE-25307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25307: -- Labels: pull-request-available (was: ) > Hive Server 2 crashes when Thrift library encounters particular security > protocol issue > --- > > Key: HIVE-25307 > URL: https://issues.apache.org/jira/browse/HIVE-25307 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Labels: pull-request-available > Attachments: hive-thrift-fix-01-3_1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > A RuntimeException is thrown by the Thrift library that causes Hive Server 2 > to crash on our customer's machine. If you Google this the exception has been > reported a couple of times over the years but not fixed. A blog (see > references below) says it is an occasional security protocol issue between > Hive Server 2 and a proxy like a Gateway. > One challenge is the Thrift TTransportFactory getTransport method declaration > throws no Exceptions hence the likely choice of RuntimeException. But that > Exception is fatal to Hive Server 2. > The proposed fix is a work around that catches RuntimeException in Hive > Server 2, saves the Exception cause in a dummy TTransport object, and throws > the cause when TTransport's open method is called later. > > ExceptionClassName: > java.lang.RuntimeException > ExceptionStackTrace: > java.lang.RuntimeException: > org.apache.thrift.transport.TSaslTransportException: No data or no sasl data > in the stream > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no > sasl data in the stream > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:326) > at > org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) > ... 10 more > > References: > [Hive server 2 thrift error - Cloudera Community - > 34293|https://community.cloudera.com/t5/Support-Questions/Hive-server-2-thrift-error/td-p/34293] > Eric Lin blog "“NO DATA OR NO SASL DATA IN THE STREAM” ERROR IN HIVESERVER2 > LOG" > [HIVE-12754] AuthTypes.NONE cause exception after HS2 start - ASF JIRA > (apache.org) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25307) Hive Server 2 crashes when Thrift library encounters particular security protocol issue
[ https://issues.apache.org/jira/browse/HIVE-25307?focusedWorklogId=619669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619669 ] ASF GitHub Bot logged work on HIVE-25307: - Author: ASF GitHub Bot Created on: 07/Jul/21 00:19 Start Date: 07/Jul/21 00:19 Worklog Time Spent: 10m Work Description: mattmccline-microsoft opened a new pull request #2451: URL: https://github.com/apache/hive/pull/2451 …icular security protocol issue ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619669) Remaining Estimate: 0h Time Spent: 10m > Hive Server 2 crashes when Thrift library encounters particular security > protocol issue > --- > > Key: HIVE-25307 > URL: https://issues.apache.org/jira/browse/HIVE-25307 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: hive-thrift-fix-01-3_1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > A RuntimeException is thrown by the Thrift library that causes Hive Server 2 > to crash on our customer's machine. If you Google this the exception has been > reported a couple of times over the years but not fixed. A blog (see > references below) says it is an occasional security protocol issue between > Hive Server 2 and a proxy like a Gateway. > One challenge is the Thrift TTransportFactory getTransport method declaration > throws no Exceptions hence the likely choice of RuntimeException. But that > Exception is fatal to Hive Server 2. > The proposed fix is a work around that catches RuntimeException in Hive > Server 2, saves the Exception cause in a dummy TTransport object, and throws > the cause when TTransport's open method is called later. > > ExceptionClassName: > java.lang.RuntimeException > ExceptionStackTrace: > java.lang.RuntimeException: > org.apache.thrift.transport.TSaslTransportException: No data or no sasl data > in the stream > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no > sasl data in the stream > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:326) > at > org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) > ... 10 more > > References: > [Hive server 2 thrift error - Cloudera Community - > 34293|https://community.cloudera.com/t5/Support-Questions/Hive-server-2-thrift-error/td-p/34293] > Eric Lin blog "“NO DATA OR NO SASL DATA IN THE STREAM” ERROR IN HIVESERVER2 > LOG" > [HIVE-12754] AuthTypes.NONE cause exception after HS2 start - ASF JIRA > (apache.org) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25282) Drop/Alter table in REMOTE db should fail
[ https://issues.apache.org/jira/browse/HIVE-25282?focusedWorklogId=619660=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619660 ] ASF GitHub Bot logged work on HIVE-25282: - Author: ASF GitHub Bot Created on: 06/Jul/21 23:47 Start Date: 06/Jul/21 23:47 Worklog Time Spent: 10m Work Description: dantongdong opened a new pull request #2450: URL: https://github.com/apache/hive/pull/2450 [HIVE-25282](https://issues.apache.org/jira/browse/HIVE-25282): Drop/Alter table in REMOTE db should fail -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619660) Remaining Estimate: 0h Time Spent: 10m > Drop/Alter table in REMOTE db should fail > - > > Key: HIVE-25282 > URL: https://issues.apache.org/jira/browse/HIVE-25282 > Project: Hive > Issue Type: Sub-task >Reporter: Dantong Dong >Assignee: Dantong Dong >Priority: Major > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Drop/Alter table statement should be explicitly rejected in REMOTE database. > In consistency with HIVE-24425: Create table in REMOTE db should fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25282) Drop/Alter table in REMOTE db should fail
[ https://issues.apache.org/jira/browse/HIVE-25282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25282: -- Labels: pull-request-available (was: ) > Drop/Alter table in REMOTE db should fail > - > > Key: HIVE-25282 > URL: https://issues.apache.org/jira/browse/HIVE-25282 > Project: Hive > Issue Type: Sub-task >Reporter: Dantong Dong >Assignee: Dantong Dong >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Drop/Alter table statement should be explicitly rejected in REMOTE database. > In consistency with HIVE-24425: Create table in REMOTE db should fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25213) Implement List getTables() for existing connectors.
[ https://issues.apache.org/jira/browse/HIVE-25213?focusedWorklogId=619589=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619589 ] ASF GitHub Bot logged work on HIVE-25213: - Author: ASF GitHub Bot Created on: 06/Jul/21 20:01 Start Date: 06/Jul/21 20:01 Worklog Time Spent: 10m Work Description: dantongdong closed pull request #2371: URL: https://github.com/apache/hive/pull/2371 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619589) Time Spent: 1h 50m (was: 1h 40m) > Implement List getTables() for existing connectors. > -- > > Key: HIVE-25213 > URL: https://issues.apache.org/jira/browse/HIVE-25213 > Project: Hive > Issue Type: Sub-task >Reporter: Naveen Gangam >Assignee: Dantong Dong >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > In the initial implementation, connector providers do not implement the > getTables(string pattern) spi. We had deferred it for later. Only > getTableNames() and getTable() were implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24970) Reject location and managed locations in DDL for REMOTE databases.
[ https://issues.apache.org/jira/browse/HIVE-24970?focusedWorklogId=619588=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619588 ] ASF GitHub Bot logged work on HIVE-24970: - Author: ASF GitHub Bot Created on: 06/Jul/21 20:01 Start Date: 06/Jul/21 20:01 Worklog Time Spent: 10m Work Description: dantongdong closed pull request #2389: URL: https://github.com/apache/hive/pull/2389 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619588) Time Spent: 1h 20m (was: 1h 10m) > Reject location and managed locations in DDL for REMOTE databases. > -- > > Key: HIVE-24970 > URL: https://issues.apache.org/jira/browse/HIVE-24970 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Dantong Dong >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > This was part of the review feedback from Yongzhi. Creating a followup jira > to track this discussion. > So, using DB connector for DB, will not create managed tables? > > @nrg4878 nrg4878 1 hour ago Author Member > we don't support create/drop/alter in REMOTE databases at this point. the > concepts of managed vs external is not in the picture at this point. When we > do support it, it will be application to the hive connectors only (or other > hive based connectors like AWS Glue) > > @nrg4878 nrg4878 2 minutes ago Author Member > will file a separate jira for this. Basically, instead of ignoring the > location and managedlocation that may be specified for remote database, the > grammer needs to not accept any locations in the DDL at all. > The argument is fair, why accept something we do not honor or entirely > irrelevant for such databases. However, this requires some thought when we > have additional connectors for remote hive instances. It might have some > relevance in terms of security with Ranger etc. > So will create new jira for followup discussion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25213) Implement List getTables() for existing connectors.
[ https://issues.apache.org/jira/browse/HIVE-25213?focusedWorklogId=619548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619548 ] ASF GitHub Bot logged work on HIVE-25213: - Author: ASF GitHub Bot Created on: 06/Jul/21 18:43 Start Date: 06/Jul/21 18:43 Worklog Time Spent: 10m Work Description: nrg4878 commented on pull request #2371: URL: https://github.com/apache/hive/pull/2371#issuecomment-874996414 Fix has been committed to master. Please close the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619548) Time Spent: 1h 40m (was: 1.5h) > Implement List getTables() for existing connectors. > -- > > Key: HIVE-25213 > URL: https://issues.apache.org/jira/browse/HIVE-25213 > Project: Hive > Issue Type: Sub-task >Reporter: Naveen Gangam >Assignee: Dantong Dong >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the initial implementation, connector providers do not implement the > getTables(string pattern) spi. We had deferred it for later. Only > getTableNames() and getTable() were implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25213) Implement List getTables() for existing connectors.
[ https://issues.apache.org/jira/browse/HIVE-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam resolved HIVE-25213. -- Fix Version/s: 4.0.0 Resolution: Fixed Fix has been committed to master. Closing the jira. Thank you for the contribution [~dantongdong] > Implement List getTables() for existing connectors. > -- > > Key: HIVE-25213 > URL: https://issues.apache.org/jira/browse/HIVE-25213 > Project: Hive > Issue Type: Sub-task >Reporter: Naveen Gangam >Assignee: Dantong Dong >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the initial implementation, connector providers do not implement the > getTables(string pattern) spi. We had deferred it for later. Only > getTableNames() and getTable() were implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25299) Casting timestamp to numeric data types is incorrect for non-UTC timezones
[
https://issues.apache.org/jira/browse/HIVE-25299?focusedWorklogId=619544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619544
]
ASF GitHub Bot logged work on HIVE-25299:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 18:37
Start Date: 06/Jul/21 18:37
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #2447:
URL: https://github.com/apache/hive/pull/2447#issuecomment-874992670
@adesh-rao Could you pls add unit tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 619544)
Time Spent: 40m (was: 0.5h)
> Casting timestamp to numeric data types is incorrect for non-UTC timezones
> --
>
> Key: HIVE-25299
> URL: https://issues.apache.org/jira/browse/HIVE-25299
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> *Hive 1.2.1*
> {noformat}
> Connected to: Apache Hive (version 1.2.1000.2.6.5.3033-1)
> Driver: Hive JDBC (version 1.2.1000.2.6.5.3033-1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 1.2.1000.2.6.5.3033-1 by Apache Hive
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as int);
> +-+--+
> | _c0 |
> +-+--+
> | 1615658400 |
> +-+--+
> 1 row selected (0.387 seconds)
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as bigint);
> +-+--+
> | _c0 |
> +-+--+
> | 1615658400 |
> +-+--+
> 1 row selected (0.369 seconds)
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as double);
> +--+--+
> | _c0 |
> +--+--+
> | 1.6156584E9 |
> +--+--+
> {noformat}
> *Hive 3.1, 4.0*
> {noformat}
> Connected to: Apache Hive (version 3.1.0.3.1.6.1-6)
> Driver: Hive JDBC (version 3.1.4.4.1.4.8)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 3.1.4.4.1.4.8 by Apache Hive
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as int);
> +-+
> | _c0 |
> +-+
> | 1615683600 |
> +-+
> 1 row selected (0.666 seconds)
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as bigint);
> +-+
> | _c0 |
> +-+
> | 1615683600 |
> +-+
> 1 row selected (0.536 seconds)
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as double);
> +--+
> | _c0 |
> +--+
> | 1.6156836E9 |
> +--+
> 1 row selected (0.696 seconds)
> {noformat}
>
> The issue occurs for non-UTC timezone (VM timezone is set to 'Asia/Bangkok').
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24970) Reject location and managed locations in DDL for REMOTE databases.
[ https://issues.apache.org/jira/browse/HIVE-24970?focusedWorklogId=619527=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619527 ] ASF GitHub Bot logged work on HIVE-24970: - Author: ASF GitHub Bot Created on: 06/Jul/21 18:05 Start Date: 06/Jul/21 18:05 Worklog Time Spent: 10m Work Description: nrg4878 commented on pull request #2389: URL: https://github.com/apache/hive/pull/2389#issuecomment-874972263 Fix has been committed. Please close the PR. Thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619527) Time Spent: 1h 10m (was: 1h) > Reject location and managed locations in DDL for REMOTE databases. > -- > > Key: HIVE-24970 > URL: https://issues.apache.org/jira/browse/HIVE-24970 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Dantong Dong >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > This was part of the review feedback from Yongzhi. Creating a followup jira > to track this discussion. > So, using DB connector for DB, will not create managed tables? > > @nrg4878 nrg4878 1 hour ago Author Member > we don't support create/drop/alter in REMOTE databases at this point. the > concepts of managed vs external is not in the picture at this point. When we > do support it, it will be application to the hive connectors only (or other > hive based connectors like AWS Glue) > > @nrg4878 nrg4878 2 minutes ago Author Member > will file a separate jira for this. Basically, instead of ignoring the > location and managedlocation that may be specified for remote database, the > grammer needs to not accept any locations in the DDL at all. > The argument is fair, why accept something we do not honor or entirely > irrelevant for such databases. However, this requires some thought when we > have additional connectors for remote hive instances. It might have some > relevance in terms of security with Ranger etc. > So will create new jira for followup discussion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24970) Reject location and managed locations in DDL for REMOTE databases.
[ https://issues.apache.org/jira/browse/HIVE-24970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam resolved HIVE-24970. -- Fix Version/s: 4.0.0 Resolution: Fixed Fix has been committed to master. Closing the jira. Thank you for the contribution [~dantongdong] > Reject location and managed locations in DDL for REMOTE databases. > -- > > Key: HIVE-24970 > URL: https://issues.apache.org/jira/browse/HIVE-24970 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Dantong Dong >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > This was part of the review feedback from Yongzhi. Creating a followup jira > to track this discussion. > So, using DB connector for DB, will not create managed tables? > > @nrg4878 nrg4878 1 hour ago Author Member > we don't support create/drop/alter in REMOTE databases at this point. the > concepts of managed vs external is not in the picture at this point. When we > do support it, it will be application to the hive connectors only (or other > hive based connectors like AWS Glue) > > @nrg4878 nrg4878 2 minutes ago Author Member > will file a separate jira for this. Basically, instead of ignoring the > location and managedlocation that may be specified for remote database, the > grammer needs to not accept any locations in the DDL at all. > The argument is fair, why accept something we do not honor or entirely > irrelevant for such databases. However, this requires some thought when we > have additional connectors for remote hive instances. It might have some > relevance in terms of security with Ranger etc. > So will create new jira for followup discussion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23931) Send ValidWriteIdList and tableId to get_*_constraints HMS APIs
[ https://issues.apache.org/jira/browse/HIVE-23931?focusedWorklogId=619519=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619519 ] ASF GitHub Bot logged work on HIVE-23931: - Author: ASF GitHub Bot Created on: 06/Jul/21 17:57 Start Date: 06/Jul/21 17:57 Worklog Time Spent: 10m Work Description: kishendas commented on pull request #2211: URL: https://github.com/apache/hive/pull/2211#issuecomment-874966970 @ashish-kumar-sharma I see that some tests are not successful. Can you please take a look ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619519) Time Spent: 2h 10m (was: 2h) > Send ValidWriteIdList and tableId to get_*_constraints HMS APIs > --- > > Key: HIVE-23931 > URL: https://issues.apache.org/jira/browse/HIVE-23931 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > Send ValidWriteIdList and tableId to get_*_constraints HMS APIs. This would > be required in order to decide whether the response should be served from the > Cache or backing DB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24499) Throw error when respective connector JDBC jar is not present in the lib/ path.
[ https://issues.apache.org/jira/browse/HIVE-24499?focusedWorklogId=619513=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619513 ] ASF GitHub Bot logged work on HIVE-24499: - Author: ASF GitHub Bot Created on: 06/Jul/21 17:43 Start Date: 06/Jul/21 17:43 Worklog Time Spent: 10m Work Description: saihemanth-cloudera closed pull request #2247: URL: https://github.com/apache/hive/pull/2247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619513) Time Spent: 0.5h (was: 20m) > Throw error when respective connector JDBC jar is not present in the lib/ > path. > --- > > Key: HIVE-24499 > URL: https://issues.apache.org/jira/browse/HIVE-24499 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24499) Throw error when respective connector JDBC jar is not present in the lib/ path.
[ https://issues.apache.org/jira/browse/HIVE-24499?focusedWorklogId=619511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619511 ] ASF GitHub Bot logged work on HIVE-24499: - Author: ASF GitHub Bot Created on: 06/Jul/21 17:39 Start Date: 06/Jul/21 17:39 Worklog Time Spent: 10m Work Description: nrg4878 commented on pull request #2247: URL: https://github.com/apache/hive/pull/2247#issuecomment-874955014 @saihemanth-cloudera Can you please close this PR? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619511) Time Spent: 20m (was: 10m) > Throw error when respective connector JDBC jar is not present in the lib/ > path. > --- > > Key: HIVE-24499 > URL: https://issues.apache.org/jira/browse/HIVE-24499 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25137) getAllWriteEventInfo should go through the HMS client instead of using RawStore directly
[
https://issues.apache.org/jira/browse/HIVE-25137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu-Wen Lai reassigned HIVE-25137:
-
Assignee: Yu-Wen Lai (was: Pratyush Madhukar)
> getAllWriteEventInfo should go through the HMS client instead of using
> RawStore directly
>
>
> Key: HIVE-25137
> URL: https://issues.apache.org/jira/browse/HIVE-25137
> Project: Hive
> Issue Type: Improvement
>Reporter: Pratyush Madhukar
>Assignee: Yu-Wen Lai
>Priority: Major
>
> {code:java}
> private List getAllWriteEventInfo(Context withinContext)
> throws Exception {
> String contextDbName =
> StringUtils.normalizeIdentifier(withinContext.replScope.getDbName());
> RawStore rawStore =
> HiveMetaStore.HMSHandler.getMSForConf(withinContext.hiveConf);
> List writeEventInfoList
> = rawStore.getAllWriteEventInfo(eventMessage.getTxnId(),
> contextDbName, null);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25299) Casting timestamp to numeric data types is incorrect for non-UTC timezones
[
https://issues.apache.org/jira/browse/HIVE-25299?focusedWorklogId=619444=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619444
]
ASF GitHub Bot logged work on HIVE-25299:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 16:08
Start Date: 06/Jul/21 16:08
Worklog Time Spent: 10m
Work Description: adesh-rao commented on pull request #2447:
URL: https://github.com/apache/hive/pull/2447#issuecomment-874891754
The jira was just to fix the timestamp conversion. Also, UDFToInteger seems
to be used for acid related code too (contains a method to convert
recordidentifier to int).
Instead of doing refactoring and fixing timestamp conversion in the same
jira, We can create a followup jira to refactor the UDF. I will create that
create and attach the link here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 619444)
Time Spent: 0.5h (was: 20m)
> Casting timestamp to numeric data types is incorrect for non-UTC timezones
> --
>
> Key: HIVE-25299
> URL: https://issues.apache.org/jira/browse/HIVE-25299
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> *Hive 1.2.1*
> {noformat}
> Connected to: Apache Hive (version 1.2.1000.2.6.5.3033-1)
> Driver: Hive JDBC (version 1.2.1000.2.6.5.3033-1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 1.2.1000.2.6.5.3033-1 by Apache Hive
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as int);
> +-+--+
> | _c0 |
> +-+--+
> | 1615658400 |
> +-+--+
> 1 row selected (0.387 seconds)
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as bigint);
> +-+--+
> | _c0 |
> +-+--+
> | 1615658400 |
> +-+--+
> 1 row selected (0.369 seconds)
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as double);
> +--+--+
> | _c0 |
> +--+--+
> | 1.6156584E9 |
> +--+--+
> {noformat}
> *Hive 3.1, 4.0*
> {noformat}
> Connected to: Apache Hive (version 3.1.0.3.1.6.1-6)
> Driver: Hive JDBC (version 3.1.4.4.1.4.8)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 3.1.4.4.1.4.8 by Apache Hive
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as int);
> +-+
> | _c0 |
> +-+
> | 1615683600 |
> +-+
> 1 row selected (0.666 seconds)
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as bigint);
> +-+
> | _c0 |
> +-+
> | 1615683600 |
> +-+
> 1 row selected (0.536 seconds)
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as double);
> +--+
> | _c0 |
> +--+
> | 1.6156836E9 |
> +--+
> 1 row selected (0.696 seconds)
> {noformat}
>
> The issue occurs for non-UTC timezone (VM timezone is set to 'Asia/Bangkok').
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25209) SELECT query with SUM function producing unexpected result
[
https://issues.apache.org/jira/browse/HIVE-25209?focusedWorklogId=619399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619399
]
ASF GitHub Bot logged work on HIVE-25209:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 15:01
Start Date: 06/Jul/21 15:01
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #2360:
URL: https://github.com/apache/hive/pull/2360#discussion_r664637117
##
File path: ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java
##
@@ -397,6 +397,11 @@ else if (getGbyKeyType(cgbyOp) == GbyKeyType.CONSTANT &&
rowCnt == 0) {
if (udaf instanceof GenericUDAFSum) {
// long/double/decimal
ExprNodeDesc desc = aggr.getParameters().get(0);
+// return null for SUM(1), when the table is empty. Without this,
category = LONG, and the result is 0
+// instead of NULL.
+if (desc instanceof ExprNodeConstantDesc && rowCnt == 0) {
+ return oneRow.add(null);
Review comment:
this `return` doesn't look right to me - it may work correctly if you
around only selecting the aggregate - but what happens if you have more?
I believe you should execute `oneRow.add(null)` ; bail out the rest of this
conditional - and let the code continue with the remaining aggregates if there
is any
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 619399)
Time Spent: 1h 10m (was: 1h)
> SELECT query with SUM function producing unexpected result
> --
>
> Key: HIVE-25209
> URL: https://issues.apache.org/jira/browse/HIVE-25209
> Project: Hive
> Issue Type: Bug
>Reporter: Soumyakanti Das
>Assignee: Soumyakanti Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Hive: SELECT query with SUM function producing unexpected result
> Problem Statement:
> {noformat}
> SELECT SUM(1) FROM t1;
> result: 0
> SELECT SUM(agg0) FROM (
> SELECT SUM(1) as agg0 FROM t1 WHERE t1.c0 UNION ALL
> SELECT SUM(1) as agg0 FROM t1 WHERE NOT (t1.c0) UNION ALL
> SELECT SUM(1) as agg0 FROM t1 WHERE (t1.c0) IS NULL
> ) as asdf;
> result: null {noformat}
> Steps to reproduce:
> {noformat}
> DROP DATABASE IF EXISTS db5 CASCADE;
> CREATE DATABASE db5;
> use db5;
> CREATE TABLE IF NOT EXISTS t1(c0 boolean, c1 boolean);
> SELECT SUM(1) FROM t1;
> -- result: 0
> SELECT SUM(agg0) FROM (
> SELECT SUM(1) as agg0 FROM t1 WHERE t1.c0 UNION ALL
> SELECT SUM(1) as agg0 FROM t1 WHERE NOT (t1.c0) UNION ALL
> SELECT SUM(1) as agg0 FROM t1 WHERE (t1.c0) IS NULL
> ) as asdf;
> -- result: null {noformat}
> Observations:
> SELECT SUM(1) as agg0 FROM t1 WHERE t1.c0 = t1.c1; – will result in null
> Similarity with postgres,
> both the queries result in null
> Similarity with Impala,
> both the queries result in null
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25299) Casting timestamp to numeric data types is incorrect for non-UTC timezones
[
https://issues.apache.org/jira/browse/HIVE-25299?focusedWorklogId=619335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619335
]
ASF GitHub Bot logged work on HIVE-25299:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 12:45
Start Date: 06/Jul/21 12:45
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on pull request #2447:
URL: https://github.com/apache/hive/pull/2447#issuecomment-874727428
@adesh-rao Please migrate all this UDF to genericUDF and use timestamp
convertor instead of adding timezone related change in each one. As timestamp
convertor already have everything required for timezone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 619335)
Time Spent: 20m (was: 10m)
> Casting timestamp to numeric data types is incorrect for non-UTC timezones
> --
>
> Key: HIVE-25299
> URL: https://issues.apache.org/jira/browse/HIVE-25299
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> *Hive 1.2.1*
> {noformat}
> Connected to: Apache Hive (version 1.2.1000.2.6.5.3033-1)
> Driver: Hive JDBC (version 1.2.1000.2.6.5.3033-1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 1.2.1000.2.6.5.3033-1 by Apache Hive
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as int);
> +-+--+
> | _c0 |
> +-+--+
> | 1615658400 |
> +-+--+
> 1 row selected (0.387 seconds)
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as bigint);
> +-+--+
> | _c0 |
> +-+--+
> | 1615658400 |
> +-+--+
> 1 row selected (0.369 seconds)
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as double);
> +--+--+
> | _c0 |
> +--+--+
> | 1.6156584E9 |
> +--+--+
> {noformat}
> *Hive 3.1, 4.0*
> {noformat}
> Connected to: Apache Hive (version 3.1.0.3.1.6.1-6)
> Driver: Hive JDBC (version 3.1.4.4.1.4.8)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 3.1.4.4.1.4.8 by Apache Hive
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as int);
> +-+
> | _c0 |
> +-+
> | 1615683600 |
> +-+
> 1 row selected (0.666 seconds)
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as bigint);
> +-+
> | _c0 |
> +-+
> | 1615683600 |
> +-+
> 1 row selected (0.536 seconds)
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as double);
> +--+
> | _c0 |
> +--+
> | 1.6156836E9 |
> +--+
> 1 row selected (0.696 seconds)
> {noformat}
>
> The issue occurs for non-UTC timezone (VM timezone is set to 'Asia/Bangkok').
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25299) Casting timestamp to numeric data types is incorrect for non-UTC timezones
[
https://issues.apache.org/jira/browse/HIVE-25299?focusedWorklogId=619319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619319
]
ASF GitHub Bot logged work on HIVE-25299:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 12:08
Start Date: 06/Jul/21 12:08
Worklog Time Spent: 10m
Work Description: adesh-rao opened a new pull request #2447:
URL: https://github.com/apache/hive/pull/2447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 619319)
Remaining Estimate: 0h
Time Spent: 10m
> Casting timestamp to numeric data types is incorrect for non-UTC timezones
> --
>
> Key: HIVE-25299
> URL: https://issues.apache.org/jira/browse/HIVE-25299
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Major
> Fix For: 4.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Hive 1.2.1*
> {noformat}
> Connected to: Apache Hive (version 1.2.1000.2.6.5.3033-1)
> Driver: Hive JDBC (version 1.2.1000.2.6.5.3033-1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 1.2.1000.2.6.5.3033-1 by Apache Hive
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as int);
> +-+--+
> | _c0 |
> +-+--+
> | 1615658400 |
> +-+--+
> 1 row selected (0.387 seconds)
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as bigint);
> +-+--+
> | _c0 |
> +-+--+
> | 1615658400 |
> +-+--+
> 1 row selected (0.369 seconds)
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as double);
> +--+--+
> | _c0 |
> +--+--+
> | 1.6156584E9 |
> +--+--+
> {noformat}
> *Hive 3.1, 4.0*
> {noformat}
> Connected to: Apache Hive (version 3.1.0.3.1.6.1-6)
> Driver: Hive JDBC (version 3.1.4.4.1.4.8)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 3.1.4.4.1.4.8 by Apache Hive
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as int);
> +-+
> | _c0 |
> +-+
> | 1615683600 |
> +-+
> 1 row selected (0.666 seconds)
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as bigint);
> +-+
> | _c0 |
> +-+
> | 1615683600 |
> +-+
> 1 row selected (0.536 seconds)
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as double);
> +--+
> | _c0 |
> +--+
> | 1.6156836E9 |
> +--+
> 1 row selected (0.696 seconds)
> {noformat}
>
> The issue occurs for non-UTC timezone (VM timezone is set to 'Asia/Bangkok').
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25299) Casting timestamp to numeric data types is incorrect for non-UTC timezones
[
https://issues.apache.org/jira/browse/HIVE-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25299:
--
Labels: pull-request-available (was: )
> Casting timestamp to numeric data types is incorrect for non-UTC timezones
> --
>
> Key: HIVE-25299
> URL: https://issues.apache.org/jira/browse/HIVE-25299
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Hive 1.2.1*
> {noformat}
> Connected to: Apache Hive (version 1.2.1000.2.6.5.3033-1)
> Driver: Hive JDBC (version 1.2.1000.2.6.5.3033-1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 1.2.1000.2.6.5.3033-1 by Apache Hive
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as int);
> +-+--+
> | _c0 |
> +-+--+
> | 1615658400 |
> +-+--+
> 1 row selected (0.387 seconds)
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as bigint);
> +-+--+
> | _c0 |
> +-+--+
> | 1615658400 |
> +-+--+
> 1 row selected (0.369 seconds)
> 0: jdbc:hive2://zk0-nikhil.ae4yqb3genuuvaozdf> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as double);
> +--+--+
> | _c0 |
> +--+--+
> | 1.6156584E9 |
> +--+--+
> {noformat}
> *Hive 3.1, 4.0*
> {noformat}
> Connected to: Apache Hive (version 3.1.0.3.1.6.1-6)
> Driver: Hive JDBC (version 3.1.4.4.1.4.8)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 3.1.4.4.1.4.8 by Apache Hive
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as int);
> +-+
> | _c0 |
> +-+
> | 1615683600 |
> +-+
> 1 row selected (0.666 seconds)
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as bigint);
> +-+
> | _c0 |
> +-+
> | 1615683600 |
> +-+
> 1 row selected (0.536 seconds)
> 0: jdbc:hive2://zk0-nikhil.usmltwlt0ncuxmbost> select cast ( cast
> ("2021-03-14 01:00:00" as timestamp) as double);
> +--+
> | _c0 |
> +--+
> | 1.6156836E9 |
> +--+
> 1 row selected (0.696 seconds)
> {noformat}
>
> The issue occurs for non-UTC timezone (VM timezone is set to 'Asia/Bangkok').
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25306) Move Date and Timestamp parsing from ResolverStyle.LENIENT to ResolverStyle.STRICT
[ https://issues.apache.org/jira/browse/HIVE-25306?focusedWorklogId=619258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619258 ] ASF GitHub Bot logged work on HIVE-25306: - Author: ASF GitHub Bot Created on: 06/Jul/21 11:38 Start Date: 06/Jul/21 11:38 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma commented on pull request #2445: URL: https://github.com/apache/hive/pull/2445#issuecomment-874472306 @zabetak Could you please review the PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619258) Time Spent: 40m (was: 0.5h) > Move Date and Timestamp parsing from ResolverStyle.LENIENT to > ResolverStyle.STRICT > -- > > Key: HIVE-25306 > URL: https://issues.apache.org/jira/browse/HIVE-25306 > Project: Hive > Issue Type: Bug > Components: Query Planning, UDF >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Description - > Currently Date.java and Timestamp.java use DateTimeFormatter for parsing to > convert the date/timpstamp from int,string,char etc to Date or Timestamp. > Default DateTimeFormatter which use ResolverStyle.LENIENT which mean date > like "1992-13-12" is converted to "2000-01-12", > Moving DateTimeFormatter which use ResolverStyle.STRICT which mean date like > "1992-13-12" is not be converted instead NULL is return. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24918) Handle failover case during Repl Dump
[
https://issues.apache.org/jira/browse/HIVE-24918?focusedWorklogId=619260=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619260
]
ASF GitHub Bot logged work on HIVE-24918:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 11:38
Start Date: 06/Jul/21 11:38
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #2121:
URL: https://github.com/apache/hive/pull/2121#discussion_r664101074
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -138,6 +141,226 @@ public void tearDown() throws Throwable {
primary.run("drop database if exists " + primaryDbName + "_extra cascade");
}
+ @Test
+ public void testFailoverDuringDump() throws Throwable {
+HiveConf primaryConf = primary.getConf();
+TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
+WarehouseInstance.Tuple dumpData = null;
+List failoverConfigs = Arrays.asList("'" +
HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+dumpData = primary.run("use " + primaryDbName)
+.run("create table t1 (id int) clustered by(id) into 3 buckets
stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("create table t2 (rank int) partitioned by (name string)
tblproperties(\"transactional\"=\"true\", " +
+"\"transactional_properties\"=\"insert_only\")")
+.dump(primaryDbName, failoverConfigs);
+
+//This dump is not failover ready as target db can be used for replication
only after first incremental load.
+FileSystem fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+Path dumpPath = new Path(dumpData.dumpLocation,
ReplUtils.REPL_HIVE_BASE_DIR);
+assertFalse(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+replica.load(replicatedDbName, primaryDbName)
+.run("use " + replicatedDbName)
+.run("show tables")
+.verifyResults(new String[]{"t1", "t2"})
+.run("repl status " + replicatedDbName)
+.verifyResult(dumpData.lastReplicationId);
+
+primary.run("use " + primaryDbName)
+.run("insert into t1 values(1)")
+.run("insert into t2 partition(name='Bob') values(11)")
+.run("insert into t2 partition(name='Carl') values(10)");
+
+/**Open transactions can be of two types:
Review comment:
nit:
/* Open transactions can be of two types:
Case 1) Txns that have not acquired HIVE LOCKS or they belong to
different db: These txns would be captured in
_failovermetadata file inside dump directory.
Case 2) Txns that have acquired HIVE LOCKS and belong to db under
replication: These txns would be aborted by hive
as a part of dump operation.
*/
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -138,6 +141,226 @@ public void tearDown() throws Throwable {
primary.run("drop database if exists " + primaryDbName + "_extra cascade");
}
+ @Test
+ public void testFailoverDuringDump() throws Throwable {
+HiveConf primaryConf = primary.getConf();
+TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
+WarehouseInstance.Tuple dumpData = null;
+List failoverConfigs = Arrays.asList("'" +
HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+dumpData = primary.run("use " + primaryDbName)
+.run("create table t1 (id int) clustered by(id) into 3 buckets
stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("create table t2 (rank int) partitioned by (name string)
tblproperties(\"transactional\"=\"true\", " +
+"\"transactional_properties\"=\"insert_only\")")
+.dump(primaryDbName, failoverConfigs);
+
+//This dump is not failover ready as target db can be used for replication
only after first incremental load.
+FileSystem fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+Path dumpPath = new Path(dumpData.dumpLocation,
ReplUtils.REPL_HIVE_BASE_DIR);
+assertFalse(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+replica.load(replicatedDbName, primaryDbName)
+.run("use " + replicatedDbName)
+.run("show tables")
+.verifyResults(new String[]{"t1", "t2"})
+.run("repl status " + replicatedDbName)
+.verifyResult(dumpData.lastReplicationId);
+
+primary.run("use " + primaryDbName)
+.run("insert into
[jira] [Resolved] (HIVE-25278) HiveProjectJoinTransposeRule may do invalid transformations with windowing expressions
[
https://issues.apache.org/jira/browse/HIVE-25278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich resolved HIVE-25278.
-
Fix Version/s: 4.0.0
Resolution: Fixed
merged into master
> HiveProjectJoinTransposeRule may do invalid transformations with windowing
> expressions
> ---
>
> Key: HIVE-25278
> URL: https://issues.apache.org/jira/browse/HIVE-25278
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> running
> {code}
> create table table1 (acct_num string, interest_rate decimal(10,7)) stored as
> orc;
> create table table2 (act_id string) stored as orc;
> CREATE TABLE temp_output AS
> SELECT act_nbr, row_num
> FROM (SELECT t2.act_id as act_nbr,
> row_number() over (PARTITION BY trim(acct_num) ORDER BY interest_rate DESC)
> AS row_num
> FROM table1 t1
> INNER JOIN table2 t2
> ON trim(acct_num) = t2.act_id) t
> WHERE t.row_num = 1;
> {code}
> may result in error like:
> {code}
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'interest_rate': (possible column names are:
> interest_rate, trim) (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21552) Remove tomcat:jasper-* from hive-service-rpc
[
https://issues.apache.org/jira/browse/HIVE-21552?focusedWorklogId=619204=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619204
]
ASF GitHub Bot logged work on HIVE-21552:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 11:31
Start Date: 06/Jul/21 11:31
Worklog Time Spent: 10m
Work Description: marton-bod opened a new pull request #2446:
URL: https://github.com/apache/hive/pull/2446
When committing Iceberg writes, currently we only have the JobID without the
vertexID, therefore we have to list the folder `/temp` first,
and parse out the full JobIDs (incl. vertexID) from the resulting folder names.
With Tez 0.10.1 released, now we have a new API we can call to acquire the
full JobID, making the file listing unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 619204)
Time Spent: 20m (was: 10m)
> Remove tomcat:jasper-* from hive-service-rpc
>
>
> Key: HIVE-21552
> URL: https://issues.apache.org/jira/browse/HIVE-21552
> Project: Hive
> Issue Type: Improvement
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{hive-service}} added these dependency. {{hive-service-rpc}} do not need
> these dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24918) Handle failover case during Repl Dump
[
https://issues.apache.org/jira/browse/HIVE-24918?focusedWorklogId=619191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619191
]
ASF GitHub Bot logged work on HIVE-24918:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 11:29
Start Date: 06/Jul/21 11:29
Worklog Time Spent: 10m
Work Description: hmangla98 commented on a change in pull request #2121:
URL: https://github.com/apache/hive/pull/2121#discussion_r664167672
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -138,6 +141,226 @@ public void tearDown() throws Throwable {
primary.run("drop database if exists " + primaryDbName + "_extra cascade");
}
+ @Test
+ public void testFailoverDuringDump() throws Throwable {
+HiveConf primaryConf = primary.getConf();
+TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
+WarehouseInstance.Tuple dumpData = null;
+List failoverConfigs = Arrays.asList("'" +
HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+dumpData = primary.run("use " + primaryDbName)
+.run("create table t1 (id int) clustered by(id) into 3 buckets
stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("create table t2 (rank int) partitioned by (name string)
tblproperties(\"transactional\"=\"true\", " +
+"\"transactional_properties\"=\"insert_only\")")
+.dump(primaryDbName, failoverConfigs);
+
+//This dump is not failover ready as target db can be used for replication
only after first incremental load.
+FileSystem fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+Path dumpPath = new Path(dumpData.dumpLocation,
ReplUtils.REPL_HIVE_BASE_DIR);
+assertFalse(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+replica.load(replicatedDbName, primaryDbName)
+.run("use " + replicatedDbName)
+.run("show tables")
+.verifyResults(new String[]{"t1", "t2"})
+.run("repl status " + replicatedDbName)
+.verifyResult(dumpData.lastReplicationId);
+
+primary.run("use " + primaryDbName)
+.run("insert into t1 values(1)")
+.run("insert into t2 partition(name='Bob') values(11)")
+.run("insert into t2 partition(name='Carl') values(10)");
+
+/**Open transactions can be of two types:
+ Case 1) Txns that belong to different db or have not acquired HIVE LOCKS:
These txns would be caught in
+ _failovermetadata file.
+ Case 2) Txns that belong to db under replication: These txns would be
aborted as part of dump operation.
+ */
+// Open 3 txns for Database which is not under replication
+int numTxnsForSecDb = 3;
+List txnsForSecDb = openTxns(numTxnsForSecDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of secondary db for 3 txns
+// t1=5 and t2=5
+Map tablesInSecDb = new HashMap<>();
+tablesInSecDb.put("t1", (long) numTxnsForSecDb);
+tablesInSecDb.put("t2", (long) numTxnsForSecDb);
+List lockIdsForSecDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName + "_extra",
+tablesInSecDb, txnHandler, txnsForSecDb, primaryConf);
+
+//Open 2 txns for Primary Db
+int numTxnsForPrimaryDb = 2;
+List txnsForPrimaryDb = openTxns(numTxnsForPrimaryDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of primary db for 2 txns
+// t1=5 and t2=5
+Map tablesInPrimaryDb = new HashMap<>();
+tablesInPrimaryDb.put("t1", (long) numTxnsForPrimaryDb + 1);
+tablesInPrimaryDb.put("t2", (long) numTxnsForPrimaryDb + 2);
+List lockIdsForPrimaryDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName,
+tablesInPrimaryDb, txnHandler, txnsForPrimaryDb, primaryConf);
+
+//Open 1 txn with no hive locks acquired
+List txnsWithNoLocks = openTxns(1, txnHandler, primaryConf);
+
+dumpData = primary.dump(primaryDbName, failoverConfigs);
+
+fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+dumpPath = new Path(dumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+assertTrue(fs.exists(new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString(;
+assertTrue(fs.exists(new Path(dumpPath,
FailoverMetaData.FAILOVER_METADATA)));
+assertTrue(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertTrue(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+FailoverMetaData failoverMD = new FailoverMetaData(dumpPath, conf);
+
+List openTxns = failoverMD.getOpenTxns();
+List txnsAborted = failoverMD.getAbortedTxns();
+assertTrue(txnsAborted.size() == 2);
+
[jira] [Work logged] (HIVE-25278) HiveProjectJoinTransposeRule may do invalid transformations with windowing expressions
[
https://issues.apache.org/jira/browse/HIVE-25278?focusedWorklogId=619124=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619124
]
ASF GitHub Bot logged work on HIVE-25278:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 11:20
Start Date: 06/Jul/21 11:20
Worklog Time Spent: 10m
Work Description: kgyrtkirk merged pull request #2423:
URL: https://github.com/apache/hive/pull/2423
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 619124)
Time Spent: 50m (was: 40m)
> HiveProjectJoinTransposeRule may do invalid transformations with windowing
> expressions
> ---
>
> Key: HIVE-25278
> URL: https://issues.apache.org/jira/browse/HIVE-25278
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> running
> {code}
> create table table1 (acct_num string, interest_rate decimal(10,7)) stored as
> orc;
> create table table2 (act_id string) stored as orc;
> CREATE TABLE temp_output AS
> SELECT act_nbr, row_num
> FROM (SELECT t2.act_id as act_nbr,
> row_number() over (PARTITION BY trim(acct_num) ORDER BY interest_rate DESC)
> AS row_num
> FROM table1 t1
> INNER JOIN table2 t2
> ON trim(acct_num) = t2.act_id) t
> WHERE t.row_num = 1;
> {code}
> may result in error like:
> {code}
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'interest_rate': (possible column names are:
> interest_rate, trim) (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25296) Replace parquet-hadoop-bundle dependency with the actual parquet modules
[ https://issues.apache.org/jira/browse/HIVE-25296?focusedWorklogId=619093=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619093 ] ASF GitHub Bot logged work on HIVE-25296: - Author: ASF GitHub Bot Created on: 06/Jul/21 11:16 Start Date: 06/Jul/21 11:16 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #2288: URL: https://github.com/apache/hive/pull/2288#issuecomment-874568565 looking at the jar contents I see * some slf4j stuff in the jar * a parquet.thrift at the root which is a bit odd to me * a thrift runtime * some jackson stuff I would be a bit worried from the above for the thrift runtime/compatibility - could you do a real world test to launch some tez containers and/or llap to see if everything is working properly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619093) Time Spent: 20m (was: 10m) > Replace parquet-hadoop-bundle dependency with the actual parquet modules > > > Key: HIVE-25296 > URL: https://issues.apache.org/jira/browse/HIVE-25296 > Project: Hive > Issue Type: Improvement >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The parquet-hadoop-bundle is not a real dependency but a mere packaging > of three parquet modules to create an uber jar. The Parquet community > created this artificial module on demand by HIVE-5783 but the > benefits if any are unclear. > On the contrary using the uber dependency has some drawbacks: > * Parquet souce code cannot be attached easily in IDEs which makes debugging > sessions cumbersome. > * Finding concrete dependencies with Parquet is not possible just by > inspecting the pom files. > * Extra maintenance cost for the Parquet community adding additional > verification steps during a release. > The goal of this JIRA is to replace the uber dependency with concrete > dependencies to the respective modules: > * parquet-common > * parquet-column > * parquet-hadoop -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25255) Support ALTER TABLE REPLACE COLUMNS for Iceberg
[
https://issues.apache.org/jira/browse/HIVE-25255?focusedWorklogId=619076=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619076
]
ASF GitHub Bot logged work on HIVE-25255:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 11:12
Start Date: 06/Jul/21 11:12
Worklog Time Spent: 10m
Work Description: szlta commented on a change in pull request #2418:
URL: https://github.com/apache/hive/pull/2418#discussion_r663922844
##
File path:
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveSchemaUtil.java
##
@@ -138,22 +137,88 @@ public static Type convert(TypeInfo typeInfo) {
}
/**
- * Produces the difference of two FieldSchema lists by only taking into
account the field name and type.
+ * Returns a SchemaDifference containing those fields which are present in
only one of the collections, as well as
+ * those fields which are present in both (in terms of the name) but their
type or comment has changed.
* @param minuendCollection Collection of fields to subtract from
* @param subtrahendCollection Collection of fields to subtract
- * @return the result list of difference
+ * @param bothDirections Whether or not to compute the missing fields from
the minuendCollection as well
+ * @return the difference between the two schemas
*/
- public static Collection schemaDifference(
- Collection minuendCollection, Collection
subtrahendCollection) {
+ public static SchemaDifference getSchemaDiff(Collection
minuendCollection,
+ Collection
subtrahendCollection, boolean bothDirections) {
+SchemaDifference difference = new SchemaDifference();
-Function unsetCommentFunc = fs -> new
FieldSchema(fs.getName(), fs.getType(), null);
-Set subtrahendWithoutComment =
-
subtrahendCollection.stream().map(unsetCommentFunc).collect(Collectors.toSet());
+for (FieldSchema first : minuendCollection) {
+ boolean found = false;
+ for (FieldSchema second : subtrahendCollection) {
+if (first.getName().equals(second.getName())) {
+ found = true;
+ if (!Objects.equals(first.getType(), second.getType())) {
+difference.typeChanged(first);
+ }
+ if (!Objects.equals(first.getComment(), second.getComment())) {
+difference.commentChanged(first);
+ }
+}
+ }
+ if (!found) {
+difference.missingFromSecond(first);
+ }
+}
+
+if (bothDirections) {
+ SchemaDifference otherWay = getSchemaDiff(subtrahendCollection,
minuendCollection, false);
+ otherWay.missingFromSecond().forEach(difference::missingFromFirst);
+}
-return minuendCollection.stream()
-.filter(fs ->
!subtrahendWithoutComment.contains(unsetCommentFunc.apply(fs))).collect(Collectors.toList());
+return difference;
}
+ public static class SchemaDifference {
+private final List missingFromFirst = new ArrayList<>();
+private final List missingFromSecond = new ArrayList<>();
+private final List typeChanged = new ArrayList<>();
+private final List commentChanged = new ArrayList<>();
+
+public List missingFromFirst() {
Review comment:
I'd rather avoid having an overload for this method that has a very
different behavior (get vs add), maybe you could rename either set of these
methods to be more descriptive.
##
File path:
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveSchemaUtil.java
##
@@ -138,22 +137,88 @@ public static Type convert(TypeInfo typeInfo) {
}
/**
- * Produces the difference of two FieldSchema lists by only taking into
account the field name and type.
+ * Returns a SchemaDifference containing those fields which are present in
only one of the collections, as well as
+ * those fields which are present in both (in terms of the name) but their
type or comment has changed.
* @param minuendCollection Collection of fields to subtract from
* @param subtrahendCollection Collection of fields to subtract
- * @return the result list of difference
+ * @param bothDirections Whether or not to compute the missing fields from
the minuendCollection as well
+ * @return the difference between the two schemas
*/
- public static Collection schemaDifference(
- Collection minuendCollection, Collection
subtrahendCollection) {
+ public static SchemaDifference getSchemaDiff(Collection
minuendCollection,
+ Collection
subtrahendCollection, boolean bothDirections) {
+SchemaDifference difference = new SchemaDifference();
-Function unsetCommentFunc = fs -> new
FieldSchema(fs.getName(), fs.getType(), null);
-Set subtrahendWithoutComment =
-
subtrahendCollection.stream().map(unsetCommentFunc).collect(Collectors.toSet());
+
[jira] [Work logged] (HIVE-25255) Support ALTER TABLE REPLACE COLUMNS for Iceberg
[
https://issues.apache.org/jira/browse/HIVE-25255?focusedWorklogId=619055=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619055
]
ASF GitHub Bot logged work on HIVE-25255:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 11:08
Start Date: 06/Jul/21 11:08
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #2418:
URL: https://github.com/apache/hive/pull/2418#discussion_r663975175
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
##
@@ -474,6 +480,43 @@ private static PartitionSpec spec(Configuration
configuration, Schema schema, Pr
}
}
+ private void handleReplaceColumns(org.apache.hadoop.hive.metastore.api.Table
hmsTable) throws MetaException {
+HiveSchemaUtil.SchemaDifference schemaDifference =
HiveSchemaUtil.getSchemaDiff(hmsTable.getSd().getCols(),
+HiveSchemaUtil.convert(icebergTable.schema()), true);
+if (!schemaDifference.isEmpty()) {
+ updateSchema = icebergTable.updateSchema();
+} else {
+ // we should get here if the user restated the exactly the existing
columns in the REPLACE COLUMNS command
+ LOG.info("Found no difference between new and old schema for ALTER TABLE
REPLACE COLUMNS for" +
+ " table: {}. There will be no Iceberg commit.",
hmsTable.getTableName());
Review comment:
Done
##
File path:
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveSchemaUtil.java
##
@@ -138,22 +137,88 @@ public static Type convert(TypeInfo typeInfo) {
}
/**
- * Produces the difference of two FieldSchema lists by only taking into
account the field name and type.
+ * Returns a SchemaDifference containing those fields which are present in
only one of the collections, as well as
+ * those fields which are present in both (in terms of the name) but their
type or comment has changed.
* @param minuendCollection Collection of fields to subtract from
* @param subtrahendCollection Collection of fields to subtract
- * @return the result list of difference
+ * @param bothDirections Whether or not to compute the missing fields from
the minuendCollection as well
+ * @return the difference between the two schemas
*/
- public static Collection schemaDifference(
- Collection minuendCollection, Collection
subtrahendCollection) {
+ public static SchemaDifference getSchemaDiff(Collection
minuendCollection,
+ Collection
subtrahendCollection, boolean bothDirections) {
+SchemaDifference difference = new SchemaDifference();
-Function unsetCommentFunc = fs -> new
FieldSchema(fs.getName(), fs.getType(), null);
-Set subtrahendWithoutComment =
-
subtrahendCollection.stream().map(unsetCommentFunc).collect(Collectors.toSet());
+for (FieldSchema first : minuendCollection) {
+ boolean found = false;
+ for (FieldSchema second : subtrahendCollection) {
+if (first.getName().equals(second.getName())) {
+ found = true;
+ if (!Objects.equals(first.getType(), second.getType())) {
+difference.typeChanged(first);
+ }
+ if (!Objects.equals(first.getComment(), second.getComment())) {
+difference.commentChanged(first);
+ }
+}
+ }
+ if (!found) {
+difference.missingFromSecond(first);
+ }
+}
+
+if (bothDirections) {
+ SchemaDifference otherWay = getSchemaDiff(subtrahendCollection,
minuendCollection, false);
+ otherWay.missingFromSecond().forEach(difference::missingFromFirst);
+}
-return minuendCollection.stream()
-.filter(fs ->
!subtrahendWithoutComment.contains(unsetCommentFunc.apply(fs))).collect(Collectors.toList());
+return difference;
}
+ public static class SchemaDifference {
+private final List missingFromFirst = new ArrayList<>();
+private final List missingFromSecond = new ArrayList<>();
+private final List typeChanged = new ArrayList<>();
+private final List commentChanged = new ArrayList<>();
+
+public List missingFromFirst() {
Review comment:
Good point, renamed the methods
##
File path:
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveSchemaUtil.java
##
@@ -138,22 +137,88 @@ public static Type convert(TypeInfo typeInfo) {
}
/**
- * Produces the difference of two FieldSchema lists by only taking into
account the field name and type.
+ * Returns a SchemaDifference containing those fields which are present in
only one of the collections, as well as
+ * those fields which are present in both (in terms of the name) but their
type or comment has changed.
* @param minuendCollection Collection of fields to subtract from
* @param subtrahendCollection
[jira] [Work logged] (HIVE-25306) Move Date and Timestamp parsing from ResolverStyle.LENIENT to ResolverStyle.STRICT
[ https://issues.apache.org/jira/browse/HIVE-25306?focusedWorklogId=619031=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619031 ] ASF GitHub Bot logged work on HIVE-25306: - Author: ASF GitHub Bot Created on: 06/Jul/21 11:04 Start Date: 06/Jul/21 11:04 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma opened a new pull request #2445: URL: https://github.com/apache/hive/pull/2445 ### What changes were proposed in this pull request? Currently Date.java and Timestamp.java use DateTimeFormatter for parsing to convert the date/timpstamp from int,string,char etc to Date or Timestamp. Default DateTimeFormatter which use ResolverStyle.LENIENT which mean date like "1992-13-12" is converted to "2000-01-12", Moving DateTimeFormatter which use ResolverStyle.STRICT which mean date like "1992-13-12" is not be converted instead NULL is return. ### Why are the changes needed? ResolverStyle.LENIENT to ResolverStyle.STRICT ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? UTs and QTs added as part of PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619031) Time Spent: 0.5h (was: 20m) > Move Date and Timestamp parsing from ResolverStyle.LENIENT to > ResolverStyle.STRICT > -- > > Key: HIVE-25306 > URL: https://issues.apache.org/jira/browse/HIVE-25306 > Project: Hive > Issue Type: Bug > Components: Query Planning, UDF >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Description - > Currently Date.java and Timestamp.java use DateTimeFormatter for parsing to > convert the date/timpstamp from int,string,char etc to Date or Timestamp. > Default DateTimeFormatter which use ResolverStyle.LENIENT which mean date > like "1992-13-12" is converted to "2000-01-12", > Moving DateTimeFormatter which use ResolverStyle.STRICT which mean date like > "1992-13-12" is not be converted instead NULL is return. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25252) All new compaction metrics should be lower case
[ https://issues.apache.org/jira/browse/HIVE-25252?focusedWorklogId=619002=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-619002 ] ASF GitHub Bot logged work on HIVE-25252: - Author: ASF GitHub Bot Created on: 06/Jul/21 10:59 Start Date: 06/Jul/21 10:59 Worklog Time Spent: 10m Work Description: klcopp merged pull request #2422: URL: https://github.com/apache/hive/pull/2422 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 619002) Time Spent: 1h (was: 50m) > All new compaction metrics should be lower case > --- > > Key: HIVE-25252 > URL: https://issues.apache.org/jira/browse/HIVE-25252 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > E.g: > compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor > compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major > compaction_cleaner_cycle_MINOR -> compaction_cleaner_cycle_minor > compaction_cleaner_cycle_MAJOR -> compaction_cleaner_cycle_major -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25278) HiveProjectJoinTransposeRule may do invalid transformations with windowing expressions
[
https://issues.apache.org/jira/browse/HIVE-25278?focusedWorklogId=618993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618993
]
ASF GitHub Bot logged work on HIVE-25278:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 10:31
Start Date: 06/Jul/21 10:31
Worklog Time Spent: 10m
Work Description: kgyrtkirk merged pull request #2423:
URL: https://github.com/apache/hive/pull/2423
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 618993)
Time Spent: 40m (was: 0.5h)
> HiveProjectJoinTransposeRule may do invalid transformations with windowing
> expressions
> ---
>
> Key: HIVE-25278
> URL: https://issues.apache.org/jira/browse/HIVE-25278
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> running
> {code}
> create table table1 (acct_num string, interest_rate decimal(10,7)) stored as
> orc;
> create table table2 (act_id string) stored as orc;
> CREATE TABLE temp_output AS
> SELECT act_nbr, row_num
> FROM (SELECT t2.act_id as act_nbr,
> row_number() over (PARTITION BY trim(acct_num) ORDER BY interest_rate DESC)
> AS row_num
> FROM table1 t1
> INNER JOIN table2 t2
> ON trim(acct_num) = t2.act_id) t
> WHERE t.row_num = 1;
> {code}
> may result in error like:
> {code}
> Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
> Invalid column reference 'interest_rate': (possible column names are:
> interest_rate, trim) (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25308) Use new Tez API to get JobID for Iceberg commits
[
https://issues.apache.org/jira/browse/HIVE-25308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Bod reassigned HIVE-25308:
-
> Use new Tez API to get JobID for Iceberg commits
>
>
> Key: HIVE-25308
> URL: https://issues.apache.org/jira/browse/HIVE-25308
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
>
> When committing Iceberg writes, currently we only have the JobID without the
> vertexID, therefore we have to list the folder {{/temp}}
> first, and parse out the full JobIDs (incl. vertexID) from the resulting
> folder names.
> With Tez 0.10.1 released, now we have a new API we can call to acquire the
> full JobID, making the file listing unnecessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21552) Remove tomcat:jasper-* from hive-service-rpc
[
https://issues.apache.org/jira/browse/HIVE-21552?focusedWorklogId=618989=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618989
]
ASF GitHub Bot logged work on HIVE-21552:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 10:11
Start Date: 06/Jul/21 10:11
Worklog Time Spent: 10m
Work Description: marton-bod opened a new pull request #2446:
URL: https://github.com/apache/hive/pull/2446
When committing Iceberg writes, currently we only have the JobID without the
vertexID, therefore we have to list the folder `/temp` first,
and parse out the full JobIDs (incl. vertexID) from the resulting folder names.
With Tez 0.10.1 released, now we have a new API we can call to acquire the
full JobID, making the file listing unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 618989)
Remaining Estimate: 0h
Time Spent: 10m
> Remove tomcat:jasper-* from hive-service-rpc
>
>
> Key: HIVE-21552
> URL: https://issues.apache.org/jira/browse/HIVE-21552
> Project: Hive
> Issue Type: Improvement
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{hive-service}} added these dependency. {{hive-service-rpc}} do not need
> these dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21552) Remove tomcat:jasper-* from hive-service-rpc
[
https://issues.apache.org/jira/browse/HIVE-21552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-21552:
--
Labels: pull-request-available (was: )
> Remove tomcat:jasper-* from hive-service-rpc
>
>
> Key: HIVE-21552
> URL: https://issues.apache.org/jira/browse/HIVE-21552
> Project: Hive
> Issue Type: Improvement
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{hive-service}} added these dependency. {{hive-service-rpc}} do not need
> these dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25261) RetryingHMSHandler should wrap the MetaException with short description of the target
[
https://issues.apache.org/jira/browse/HIVE-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng updated HIVE-25261:
---
Description:
[RetryingMetaStoreClient|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java#L267-L276]
relies on the message of MetaException to make decision on retrying the
current operation when failed. However the RetryingHMSHandler only wraps the
message into MetaException, which may cause the client unable to retry with
other metastore instances.
For example, if we got exception:
{noformat}
Caused by: javax.jdo.JDOFatalUserException: Persistence Manager has been closed
at
org.datanucleus.api.jdo.JDOPersistenceManager.assertIsOpen(JDOPersistenceManager.java:2235)
at
org.datanucleus.api.jdo.JDOPersistenceManager.evictAll(JDOPersistenceManager.java:481)
at
org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:635)
at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1415)
at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498){noformat}
RetryingHMSHandler will throw MetaException with message 'Persistence Manager
has been closed', which not in the recoverable pattern defined in client.
was:
[RetryingMetaStoreClient|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java#L267-L276]
relies on the message of MetaException to make decision on retrying the
current operation when failed. However the RetryingHMSHandler only wraps the
message into MetaException, which may cause the client unable to retry with
other metastore instances.
For example, if we got exception:
{code:java}
Caused by: javax.jdo.JDOFatalUserException: Persistence Manager has been closed
at
org.datanucleus.api.jdo.JDOPersistenceManager.assertIsOpen(JDOPersistenceManager.java:2235)
at
org.datanucleus.api.jdo.JDOPersistenceManager.evictAll(JDOPersistenceManager.java:481)
at
org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:635)
at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1415)
at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498){code}
RetryingHMSHandler will throw MetaException with message 'Persistence Manager
has been closed', which not in the recoverable pattern defined in client.
> RetryingHMSHandler should wrap the MetaException with short description of
> the target
> -
>
> Key: HIVE-25261
> URL: https://issues.apache.org/jira/browse/HIVE-25261
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> [RetryingMetaStoreClient|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java#L267-L276]
> relies on the message of MetaException to make decision on retrying the
> current operation when failed. However the RetryingHMSHandler only wraps the
> message into MetaException, which may cause the client unable to retry with
> other metastore instances.
> For example, if we got exception:
> {noformat}
> Caused by: javax.jdo.JDOFatalUserException: Persistence Manager has been
> closed
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.assertIsOpen(JDOPersistenceManager.java:2235)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.evictAll(JDOPersistenceManager.java:481)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:635)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1415)
> at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498){noformat}
> RetryingHMSHandler will throw MetaException with message 'Persistence Manager
> has been closed', which not in the recoverable pattern defined in client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-24858) UDFClassLoader leak in Configuration.CACHE_CLASSES
[
https://issues.apache.org/jira/browse/HIVE-24858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage resolved HIVE-24858.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Committed to master on March 10. Thanks for reviewing [~pvary]!
> UDFClassLoader leak in Configuration.CACHE_CLASSES
> --
>
> Key: HIVE-24858
> URL: https://issues.apache.org/jira/browse/HIVE-24858
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> If a UDF jar has been registered in a session and a temporary function
> created from it, when the session is closed its UDFClassLoader is not GC'd as
> it has been leaked to the session's HiveConf object's cache. Since the
> ClassLoader is not GC'd, the UDF jar's classes aren't GC'd from Metaspace.
> This can potentially lead to Metaspace OOM.
> Path to GC root is:
> {code:java}
> Class Name
> | Shallow Heap |
> Retained Heap
> ---
> contextClassLoader org.apache.hive.service.server.ThreadWithGarbageCleanup @
> 0x7164deb50 HiveServer2-Handler-Pool: Thread-72 Thread| 128 |
> 79,072
> referent java.util.WeakHashMap$Entry @ 0x7164e67d0
> | 40 |
>824
> '- [6] java.util.WeakHashMap$Entry[16] @ 0x71581aac0
> | 80 |
> 5,056
>'- table java.util.WeakHashMap @ 0x71580f510
> | 48 |
> 6,920
> '- CACHE_CLASSES class org.apache.hadoop.conf.Configuration @
> 0x71580f3d8 |
> 64 |74,528
> ---
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23924) Flaky test TestStatsReplicationScenariosACID.testForParallelBootstrapLoad
[ https://issues.apache.org/jira/browse/HIVE-23924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena reassigned HIVE-23924: --- Assignee: Ayush Saxena > Flaky test TestStatsReplicationScenariosACID.testForParallelBootstrapLoad > - > > Key: HIVE-23924 > URL: https://issues.apache.org/jira/browse/HIVE-23924 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Ayush Saxena >Priority: Major > > http://ci.hive.apache.org/job/hive-precommit/job/master/122/testReport/org.apache.hadoop.hive.ql.parse/TestStatsReplicationScenariosACID/Testing___split_08___Archive___testForParallelBootstrapLoad/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25296) Replace parquet-hadoop-bundle dependency with the actual parquet modules
[ https://issues.apache.org/jira/browse/HIVE-25296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25296: -- Labels: pull-request-available (was: ) > Replace parquet-hadoop-bundle dependency with the actual parquet modules > > > Key: HIVE-25296 > URL: https://issues.apache.org/jira/browse/HIVE-25296 > Project: Hive > Issue Type: Improvement >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The parquet-hadoop-bundle is not a real dependency but a mere packaging > of three parquet modules to create an uber jar. The Parquet community > created this artificial module on demand by HIVE-5783 but the > benefits if any are unclear. > On the contrary using the uber dependency has some drawbacks: > * Parquet souce code cannot be attached easily in IDEs which makes debugging > sessions cumbersome. > * Finding concrete dependencies with Parquet is not possible just by > inspecting the pom files. > * Extra maintenance cost for the Parquet community adding additional > verification steps during a release. > The goal of this JIRA is to replace the uber dependency with concrete > dependencies to the respective modules: > * parquet-common > * parquet-column > * parquet-hadoop -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25296) Replace parquet-hadoop-bundle dependency with the actual parquet modules
[ https://issues.apache.org/jira/browse/HIVE-25296?focusedWorklogId=618956=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618956 ] ASF GitHub Bot logged work on HIVE-25296: - Author: ASF GitHub Bot Created on: 06/Jul/21 08:31 Start Date: 06/Jul/21 08:31 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #2288: URL: https://github.com/apache/hive/pull/2288#issuecomment-874568565 looking at the jar contents I see * some slf4j stuff in the jar * a parquet.thrift at the root which is a bit odd to me * a thrift runtime * some jackson stuff I would be a bit worried from the above for the thrift runtime/compatibility - could you do a real world test to launch some tez containers and/or llap to see if everything is working properly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 618956) Remaining Estimate: 0h Time Spent: 10m > Replace parquet-hadoop-bundle dependency with the actual parquet modules > > > Key: HIVE-25296 > URL: https://issues.apache.org/jira/browse/HIVE-25296 > Project: Hive > Issue Type: Improvement >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The parquet-hadoop-bundle is not a real dependency but a mere packaging > of three parquet modules to create an uber jar. The Parquet community > created this artificial module on demand by HIVE-5783 but the > benefits if any are unclear. > On the contrary using the uber dependency has some drawbacks: > * Parquet souce code cannot be attached easily in IDEs which makes debugging > sessions cumbersome. > * Finding concrete dependencies with Parquet is not possible just by > inspecting the pom files. > * Extra maintenance cost for the Parquet community adding additional > verification steps during a release. > The goal of this JIRA is to replace the uber dependency with concrete > dependencies to the respective modules: > * parquet-common > * parquet-column > * parquet-hadoop -- This message was sent by Atlassian Jira (v8.3.4#803005)
