[jira] [Work logged] (HIVE-24426) Spark job fails with fixed LlapTaskUmbilicalServer port
[
https://issues.apache.org/jira/browse/HIVE-24426?focusedWorklogId=516494=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516494
]
ASF GitHub Bot logged work on HIVE-24426:
-
Author: ASF GitHub Bot
Created on: 25/Nov/20 06:05
Start Date: 25/Nov/20 06:05
Worklog Time Spent: 10m
Work Description: ayushtkn opened a new pull request #1705:
URL: https://github.com/apache/hive/pull/1705
https://issues.apache.org/jira/browse/HIVE-24426
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516494)
Remaining Estimate: 0h
Time Spent: 10m
> Spark job fails with fixed LlapTaskUmbilicalServer port
> ---
>
> Key: HIVE-24426
> URL: https://issues.apache.org/jira/browse/HIVE-24426
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Critical
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case of cloud deployments, multiple executors are launched on name node,
> and incase a fixed umbilical port is specified using
> {{spark.hadoop.hive.llap.daemon.umbilical.port=30006}}
> The job fails with BindException.
> {noformat}
> Caused by: java.net.BindException: Problem binding to [0.0.0.0:30006]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:840)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
> at org.apache.hadoop.ipc.Server.bind(Server.java:605)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:1169)
> at org.apache.hadoop.ipc.Server.(Server.java:3032)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:1039)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Server.(WritableRpcEngine.java:438)
> at
> org.apache.hadoop.ipc.WritableRpcEngine.getServer(WritableRpcEngine.java:332)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:848)
> at
> org.apache.hadoop.hive.llap.tezplugins.helpers.LlapTaskUmbilicalServer.(LlapTaskUmbilicalServer.java:67)
> at
> org.apache.hadoop.hive.llap.ext.LlapTaskUmbilicalExternalClient$SharedUmbilicalServer.(LlapTaskUmbilicalExternalClient.java:122)
> ... 26 more
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
> at org.apache.hadoop.ipc.Server.bind(Server.java:588)
> ... 34 more{noformat}
> To counter this, better to provide a range of ports
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24426) Spark job fails with fixed LlapTaskUmbilicalServer port
[
https://issues.apache.org/jira/browse/HIVE-24426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24426:
--
Labels: pull-request-available (was: )
> Spark job fails with fixed LlapTaskUmbilicalServer port
> ---
>
> Key: HIVE-24426
> URL: https://issues.apache.org/jira/browse/HIVE-24426
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case of cloud deployments, multiple executors are launched on name node,
> and incase a fixed umbilical port is specified using
> {{spark.hadoop.hive.llap.daemon.umbilical.port=30006}}
> The job fails with BindException.
> {noformat}
> Caused by: java.net.BindException: Problem binding to [0.0.0.0:30006]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:840)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
> at org.apache.hadoop.ipc.Server.bind(Server.java:605)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:1169)
> at org.apache.hadoop.ipc.Server.(Server.java:3032)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:1039)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Server.(WritableRpcEngine.java:438)
> at
> org.apache.hadoop.ipc.WritableRpcEngine.getServer(WritableRpcEngine.java:332)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:848)
> at
> org.apache.hadoop.hive.llap.tezplugins.helpers.LlapTaskUmbilicalServer.(LlapTaskUmbilicalServer.java:67)
> at
> org.apache.hadoop.hive.llap.ext.LlapTaskUmbilicalExternalClient$SharedUmbilicalServer.(LlapTaskUmbilicalExternalClient.java:122)
> ... 26 more
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
> at org.apache.hadoop.ipc.Server.bind(Server.java:588)
> ... 34 more{noformat}
> To counter this, better to provide a range of ports
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24426) Spark job fails with fixed LlapTaskUmbilicalServer port
[
https://issues.apache.org/jira/browse/HIVE-24426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-24426:
Parent: HIVE-24058
Issue Type: Sub-task (was: Bug)
> Spark job fails with fixed LlapTaskUmbilicalServer port
> ---
>
> Key: HIVE-24426
> URL: https://issues.apache.org/jira/browse/HIVE-24426
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Critical
>
> In case of cloud deployments, multiple executors are launched on name node,
> and incase a fixed umbilical port is specified using
> {{spark.hadoop.hive.llap.daemon.umbilical.port=30006}}
> The job fails with BindException.
> {noformat}
> Caused by: java.net.BindException: Problem binding to [0.0.0.0:30006]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:840)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
> at org.apache.hadoop.ipc.Server.bind(Server.java:605)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:1169)
> at org.apache.hadoop.ipc.Server.(Server.java:3032)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:1039)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Server.(WritableRpcEngine.java:438)
> at
> org.apache.hadoop.ipc.WritableRpcEngine.getServer(WritableRpcEngine.java:332)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:848)
> at
> org.apache.hadoop.hive.llap.tezplugins.helpers.LlapTaskUmbilicalServer.(LlapTaskUmbilicalServer.java:67)
> at
> org.apache.hadoop.hive.llap.ext.LlapTaskUmbilicalExternalClient$SharedUmbilicalServer.(LlapTaskUmbilicalExternalClient.java:122)
> ... 26 more
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
> at org.apache.hadoop.ipc.Server.bind(Server.java:588)
> ... 34 more{noformat}
> To counter this, better to provide a range of ports
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24426) Spark job fails with fixed LlapTaskUmbilicalServer port
[
https://issues.apache.org/jira/browse/HIVE-24426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-24426:
Summary: Spark job fails with fixed LlapTaskUmbilicalServer port (was:
Spark job fails with fixed umbilical server port)
> Spark job fails with fixed LlapTaskUmbilicalServer port
> ---
>
> Key: HIVE-24426
> URL: https://issues.apache.org/jira/browse/HIVE-24426
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Critical
>
> In case of cloud deployments, multiple executors are launched on name node,
> and incase a fixed umbilical port is specified using
> {{spark.hadoop.hive.llap.daemon.umbilical.port=30006}}
> The job fails with BindException.
> {noformat}
> Caused by: java.net.BindException: Problem binding to [0.0.0.0:30006]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:840)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
> at org.apache.hadoop.ipc.Server.bind(Server.java:605)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:1169)
> at org.apache.hadoop.ipc.Server.(Server.java:3032)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:1039)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Server.(WritableRpcEngine.java:438)
> at
> org.apache.hadoop.ipc.WritableRpcEngine.getServer(WritableRpcEngine.java:332)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:848)
> at
> org.apache.hadoop.hive.llap.tezplugins.helpers.LlapTaskUmbilicalServer.(LlapTaskUmbilicalServer.java:67)
> at
> org.apache.hadoop.hive.llap.ext.LlapTaskUmbilicalExternalClient$SharedUmbilicalServer.(LlapTaskUmbilicalExternalClient.java:122)
> ... 26 more
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
> at org.apache.hadoop.ipc.Server.bind(Server.java:588)
> ... 34 more{noformat}
> To counter this, better to provide a range of ports
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24426) Spark job fails with fixed umbilical server port
[
https://issues.apache.org/jira/browse/HIVE-24426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena reassigned HIVE-24426:
---
> Spark job fails with fixed umbilical server port
>
>
> Key: HIVE-24426
> URL: https://issues.apache.org/jira/browse/HIVE-24426
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Critical
>
> In case of cloud deployments, multiple executors are launched on name node,
> and incase a fixed umbilical port is specified using
> {{spark.hadoop.hive.llap.daemon.umbilical.port=30006}}
> The job fails with BindException.
> {noformat}
> Caused by: java.net.BindException: Problem binding to [0.0.0.0:30006]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:840)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
> at org.apache.hadoop.ipc.Server.bind(Server.java:605)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:1169)
> at org.apache.hadoop.ipc.Server.(Server.java:3032)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:1039)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Server.(WritableRpcEngine.java:438)
> at
> org.apache.hadoop.ipc.WritableRpcEngine.getServer(WritableRpcEngine.java:332)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:848)
> at
> org.apache.hadoop.hive.llap.tezplugins.helpers.LlapTaskUmbilicalServer.(LlapTaskUmbilicalServer.java:67)
> at
> org.apache.hadoop.hive.llap.ext.LlapTaskUmbilicalExternalClient$SharedUmbilicalServer.(LlapTaskUmbilicalExternalClient.java:122)
> ... 26 more
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
> at org.apache.hadoop.ipc.Server.bind(Server.java:588)
> ... 34 more{noformat}
> To counter this, better to provide a range of ports
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-24412) Extract common parts of HiveDeCorrelator
[ https://issues.apache.org/jira/browse/HIVE-24412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa resolved HIVE-24412. --- Resolution: Fixed Pushed to master. Thanks [~vgarg], [~mgergely] for review. > Extract common parts of HiveDeCorrelator > > > Key: HIVE-24412 > URL: https://issues.apache.org/jira/browse/HIVE-24412 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 4.0.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Some methods in *HiveRelDecorrelator.decorrelateRel()* has a Calcite and a > Hive version and the implementation is the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24412) Extract common parts of HiveDeCorrelator
[ https://issues.apache.org/jira/browse/HIVE-24412?focusedWorklogId=516491=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516491 ] ASF GitHub Bot logged work on HIVE-24412: - Author: ASF GitHub Bot Created on: 25/Nov/20 05:37 Start Date: 25/Nov/20 05:37 Worklog Time Spent: 10m Work Description: kasakrisz merged pull request #1697: URL: https://github.com/apache/hive/pull/1697 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516491) Time Spent: 0.5h (was: 20m) > Extract common parts of HiveDeCorrelator > > > Key: HIVE-24412 > URL: https://issues.apache.org/jira/browse/HIVE-24412 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 4.0.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Some methods in *HiveRelDecorrelator.decorrelateRel()* has a Calcite and a > Hive version and the implementation is the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24422) Throw SemanticException when CTE alias is conflicted with table name
[
https://issues.apache.org/jira/browse/HIVE-24422?focusedWorklogId=516490=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516490
]
ASF GitHub Bot logged work on HIVE-24422:
-
Author: ASF GitHub Bot
Created on: 25/Nov/20 05:26
Start Date: 25/Nov/20 05:26
Worklog Time Spent: 10m
Work Description: dengzhhu653 closed pull request #1702:
URL: https://github.com/apache/hive/pull/1702
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516490)
Time Spent: 0.5h (was: 20m)
> Throw SemanticException when CTE alias is conflicted with table name
>
>
> Key: HIVE-24422
> URL: https://issues.apache.org/jira/browse/HIVE-24422
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> If the alias of CTE is conflicted with the table name, we use the alias
> fetching the table other than replacing it with the ASTNode tree, this may
> cause some confusing problems. For example:
> {noformat}
> create table game_info (game_name string);
> with game_info as (
> select distinct ext_id, dev_app_id, game_name
> from game_info_extend )
> select count(game_name) from game_info;{noformat}
> The query will return the number of rows of the table game_info, instead of
> the game_info_extend. Maybe we should better throw an exception to avoid such
> cases.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24408) Upgrade Parquet to 1.11.1
[ https://issues.apache.org/jira/browse/HIVE-24408?focusedWorklogId=516478=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516478 ] ASF GitHub Bot logged work on HIVE-24408: - Author: ASF GitHub Bot Created on: 25/Nov/20 04:37 Start Date: 25/Nov/20 04:37 Worklog Time Spent: 10m Work Description: dongjoon-hyun commented on pull request #1692: URL: https://github.com/apache/hive/pull/1692#issuecomment-733458649 Thank you, @jcamachor and @sunchao ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516478) Time Spent: 1.5h (was: 1h 20m) > Upgrade Parquet to 1.11.1 > - > > Key: HIVE-24408 > URL: https://issues.apache.org/jira/browse/HIVE-24408 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Parquet 1.11.1 has some bug fixes so Hive should consider to upgrade to it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24422) Throw SemanticException when CTE alias is conflicted with table name
[
https://issues.apache.org/jira/browse/HIVE-24422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng resolved HIVE-24422.
Resolution: Invalid
> Throw SemanticException when CTE alias is conflicted with table name
>
>
> Key: HIVE-24422
> URL: https://issues.apache.org/jira/browse/HIVE-24422
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> If the alias of CTE is conflicted with the table name, we use the alias
> fetching the table other than replacing it with the ASTNode tree, this may
> cause some confusing problems. For example:
> {noformat}
> create table game_info (game_name string);
> with game_info as (
> select distinct ext_id, dev_app_id, game_name
> from game_info_extend )
> select count(game_name) from game_info;{noformat}
> The query will return the number of rows of the table game_info, instead of
> the game_info_extend. Maybe we should better throw an exception to avoid such
> cases.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HIVE-24422) Throw SemanticException when CTE alias is conflicted with table name
[
https://issues.apache.org/jira/browse/HIVE-24422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng reopened HIVE-24422:
> Throw SemanticException when CTE alias is conflicted with table name
>
>
> Key: HIVE-24422
> URL: https://issues.apache.org/jira/browse/HIVE-24422
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> If the alias of CTE is conflicted with the table name, we use the alias
> fetching the table other than replacing it with the ASTNode tree, this may
> cause some confusing problems. For example:
> {noformat}
> create table game_info (game_name string);
> with game_info as (
> select distinct ext_id, dev_app_id, game_name
> from game_info_extend )
> select count(game_name) from game_info;{noformat}
> The query will return the number of rows of the table game_info, instead of
> the game_info_extend. Maybe we should better throw an exception to avoid such
> cases.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-24422) Throw SemanticException when CTE alias is conflicted with table name
[
https://issues.apache.org/jira/browse/HIVE-24422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng resolved HIVE-24422.
Resolution: Invalid
> Throw SemanticException when CTE alias is conflicted with table name
>
>
> Key: HIVE-24422
> URL: https://issues.apache.org/jira/browse/HIVE-24422
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> If the alias of CTE is conflicted with the table name, we use the alias
> fetching the table other than replacing it with the ASTNode tree, this may
> cause some confusing problems. For example:
> {noformat}
> create table game_info (game_name string);
> with game_info as (
> select distinct ext_id, dev_app_id, game_name
> from game_info_extend )
> select count(game_name) from game_info;{noformat}
> The query will return the number of rows of the table game_info, instead of
> the game_info_extend. Maybe we should better throw an exception to avoid such
> cases.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24422) Throw SemanticException when CTE alias is conflicted with table name
[
https://issues.apache.org/jira/browse/HIVE-24422?focusedWorklogId=516443=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516443
]
ASF GitHub Bot logged work on HIVE-24422:
-
Author: ASF GitHub Bot
Created on: 25/Nov/20 02:38
Start Date: 25/Nov/20 02:38
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on pull request #1702:
URL: https://github.com/apache/hive/pull/1702#issuecomment-733425360
Seems I am wrong. In the trunk, the query
`with game_info as (
select distinct ext_id, dev_app_id, game_name
from game_info_extend )
select count(game_name) from game_info;` , the game_info is firstly
materialized, this can reduce recomputing of game_info no matter how many times
the game_info is referenced later on.
The condition `if (tab == null ||
tab.getDbName().equals(SessionState.get().getCurrentDatabase())) `
in getMetaData will pass as the latter predicate turns out to be true, so the
materialized view is taken other than the table `game_info` for latter use.
@jcamachor, Am I right?
Thanks in advance.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516443)
Time Spent: 20m (was: 10m)
> Throw SemanticException when CTE alias is conflicted with table name
>
>
> Key: HIVE-24422
> URL: https://issues.apache.org/jira/browse/HIVE-24422
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> If the alias of CTE is conflicted with the table name, we use the alias
> fetching the table other than replacing it with the ASTNode tree, this may
> cause some confusing problems. For example:
> {noformat}
> create table game_info (game_name string);
> with game_info as (
> select distinct ext_id, dev_app_id, game_name
> from game_info_extend )
> select count(game_name) from game_info;{noformat}
> The query will return the number of rows of the table game_info, instead of
> the game_info_extend. Maybe we should better throw an exception to avoid such
> cases.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23980) Shade guava from existing Hive versions
[
https://issues.apache.org/jira/browse/HIVE-23980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238436#comment-17238436
]
Chao Sun commented on HIVE-23980:
-
Re-open this because there are some test failures related. The change was
reverted from branch-2 and branch-2.3. [~viirya] is working on a new version in
the PR #1356.
> Shade guava from existing Hive versions
> ---
>
> Key: HIVE-23980
> URL: https://issues.apache.org/jira/browse/HIVE-23980
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.7
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23980.01.branch-2.3.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> I'm trying to upgrade Guava version in Spark. The JIRA ticket is SPARK-32502.
> Running test hits an error:
> {code}
> sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError:
> tried to access method
> com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator;
> from class org.apache.hadoop.hive.ql.exec.FetchOperator
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> {code}
> I know that hive-exec doesn't shade Guava until HIVE-22126 but that work
> targets 4.0.0. I'm wondering if there is a solution for current Hive
> versions, e.g. Hive 2.3.7? Any ideas?
> Thanks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23980) Shade guava from existing Hive versions
[
https://issues.apache.org/jira/browse/HIVE-23980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-23980:
Fix Version/s: (was: 2.4.0)
> Shade guava from existing Hive versions
> ---
>
> Key: HIVE-23980
> URL: https://issues.apache.org/jira/browse/HIVE-23980
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.7
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23980.01.branch-2.3.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> I'm trying to upgrade Guava version in Spark. The JIRA ticket is SPARK-32502.
> Running test hits an error:
> {code}
> sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError:
> tried to access method
> com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator;
> from class org.apache.hadoop.hive.ql.exec.FetchOperator
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> {code}
> I know that hive-exec doesn't shade Guava until HIVE-22126 but that work
> targets 4.0.0. I'm wondering if there is a solution for current Hive
> versions, e.g. Hive 2.3.7? Any ideas?
> Thanks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HIVE-23980) Shade guava from existing Hive versions
[
https://issues.apache.org/jira/browse/HIVE-23980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun reopened HIVE-23980:
-
> Shade guava from existing Hive versions
> ---
>
> Key: HIVE-23980
> URL: https://issues.apache.org/jira/browse/HIVE-23980
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.7
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.4.0
>
> Attachments: HIVE-23980.01.branch-2.3.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> I'm trying to upgrade Guava version in Spark. The JIRA ticket is SPARK-32502.
> Running test hits an error:
> {code}
> sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError:
> tried to access method
> com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator;
> from class org.apache.hadoop.hive.ql.exec.FetchOperator
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> {code}
> I know that hive-exec doesn't shade Guava until HIVE-22126 but that work
> targets 4.0.0. I'm wondering if there is a solution for current Hive
> versions, e.g. Hive 2.3.7? Any ideas?
> Thanks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24324) Remove deprecated API usage from Avro
[
https://issues.apache.org/jira/browse/HIVE-24324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-24324:
Fix Version/s: 2.3.8
> Remove deprecated API usage from Avro
> -
>
> Key: HIVE-24324
> URL: https://issues.apache.org/jira/browse/HIVE-24324
> Project: Hive
> Issue Type: Improvement
> Components: Avro
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.8, 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {{JsonProperties#getJsonProp}} has been marked as deprecated in Avro 1.8 and
> removed since Avro 1.9. This replaces the API usage for this with
> {{getObjectProp}} which doesn't leak Json node from jackson. This will help
> downstream apps to depend on Hive while using higher version of Avro, and
> also help Hive to upgrade Avro version itself.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24379) Backport HIVE-19662 to branch-2.3
[ https://issues.apache.org/jira/browse/HIVE-24379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-24379: Fix Version/s: 2.3.8 > Backport HIVE-19662 to branch-2.3 > - > > Key: HIVE-24379 > URL: https://issues.apache.org/jira/browse/HIVE-24379 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Major > Labels: pull-request-available > Fix For: 2.3.8 > > Time Spent: 40m > Remaining Estimate: 0h > > In order to backport HIVE-24324 to branch-2.3 to remove deprecated Avro API > so that downstream applications who depend on Hive 2.3.x can upgrade their > Avro version, we'll need to first backport HIVE-19662 and bump Avro version > in branch-2.3 first as it is currently on 1.7.x, while HIVE-24324 requires > Avro 1.8.x. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23323) Add qsplits profile
[ https://issues.apache.org/jira/browse/HIVE-23323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-23323: Fix Version/s: 2.3.8 > Add qsplits profile > --- > > Key: HIVE-23323 > URL: https://issues.apache.org/jira/browse/HIVE-23323 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Fix For: 2.3.8, 4.0.0 > > Attachments: HIVE-23323.01.patch, HIVE-23323.02.patch, > HIVE-23323.02.patch, HIVE-23323.02.patch, HIVE-23323.03.patch > > Time Spent: 50m > Remaining Estimate: 0h > > provide an in-maven way to split-up big testclasses into smaller ones -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24035) Add Jenkinsfile for branch-2.3
[ https://issues.apache.org/jira/browse/HIVE-24035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HIVE-24035. - Fix Version/s: 2.3.8 Hadoop Flags: Reviewed Resolution: Fixed > Add Jenkinsfile for branch-2.3 > -- > > Key: HIVE-24035 > URL: https://issues.apache.org/jira/browse/HIVE-24035 > Project: Hive > Issue Type: Test >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Major > Labels: pull-request-available > Fix For: 2.3.8 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > To enable precommit tests for github PR, we need to have a Jenkinsfile in the > repo. This is already done for master and branch-2. This adds the same for > branch-2.3 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24051) Hive lineage information exposed in ExecuteWithHookContext
[ https://issues.apache.org/jira/browse/HIVE-24051?focusedWorklogId=516420=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516420 ] ASF GitHub Bot logged work on HIVE-24051: - Author: ASF GitHub Bot Created on: 25/Nov/20 01:03 Start Date: 25/Nov/20 01:03 Worklog Time Spent: 10m Work Description: sunchao commented on pull request #1413: URL: https://github.com/apache/hive/pull/1413#issuecomment-733396811 Merged to trunk. Thanks @szehon-ho ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516420) Time Spent: 1h 40m (was: 1.5h) > Hive lineage information exposed in ExecuteWithHookContext > -- > > Key: HIVE-24051 > URL: https://issues.apache.org/jira/browse/HIVE-24051 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24051.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The lineage information is not populated unless certain hooks are enabled. > However, this is a bit fragile, and no way for another hook that we write to > get this information. This proposes a flag to enable this instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24051) Hive lineage information exposed in ExecuteWithHookContext
[ https://issues.apache.org/jira/browse/HIVE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-24051: Fix Version/s: 4.0.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > Hive lineage information exposed in ExecuteWithHookContext > -- > > Key: HIVE-24051 > URL: https://issues.apache.org/jira/browse/HIVE-24051 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-24051.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The lineage information is not populated unless certain hooks are enabled. > However, this is a bit fragile, and no way for another hook that we write to > get this information. This proposes a flag to enable this instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24051) Hive lineage information exposed in ExecuteWithHookContext
[ https://issues.apache.org/jira/browse/HIVE-24051?focusedWorklogId=516419=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516419 ] ASF GitHub Bot logged work on HIVE-24051: - Author: ASF GitHub Bot Created on: 25/Nov/20 01:03 Start Date: 25/Nov/20 01:03 Worklog Time Spent: 10m Work Description: sunchao merged pull request #1413: URL: https://github.com/apache/hive/pull/1413 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516419) Time Spent: 1.5h (was: 1h 20m) > Hive lineage information exposed in ExecuteWithHookContext > -- > > Key: HIVE-24051 > URL: https://issues.apache.org/jira/browse/HIVE-24051 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24051.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > The lineage information is not populated unless certain hooks are enabled. > However, this is a bit fragile, and no way for another hook that we write to > get this information. This proposes a flag to enable this instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24051) Hive lineage information exposed in ExecuteWithHookContext
[ https://issues.apache.org/jira/browse/HIVE-24051?focusedWorklogId=516421=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516421 ] ASF GitHub Bot logged work on HIVE-24051: - Author: ASF GitHub Bot Created on: 25/Nov/20 01:03 Start Date: 25/Nov/20 01:03 Worklog Time Spent: 10m Work Description: sunchao edited a comment on pull request #1413: URL: https://github.com/apache/hive/pull/1413#issuecomment-733396811 Merged to master. Thanks @szehon-ho ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516421) Time Spent: 1h 50m (was: 1h 40m) > Hive lineage information exposed in ExecuteWithHookContext > -- > > Key: HIVE-24051 > URL: https://issues.apache.org/jira/browse/HIVE-24051 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-24051.patch > > Time Spent: 1h 50m > Remaining Estimate: 0h > > The lineage information is not populated unless certain hooks are enabled. > However, this is a bit fragile, and no way for another hook that we write to > get this information. This proposes a flag to enable this instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24135) Drop database doesn't delete directory in managed location
[
https://issues.apache.org/jira/browse/HIVE-24135?focusedWorklogId=516412=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516412
]
ASF GitHub Bot logged work on HIVE-24135:
-
Author: ASF GitHub Bot
Created on: 25/Nov/20 00:43
Start Date: 25/Nov/20 00:43
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #1506:
URL: https://github.com/apache/hive/pull/1506
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516412)
Time Spent: 0.5h (was: 20m)
> Drop database doesn't delete directory in managed location
> --
>
> Key: HIVE-24135
> URL: https://issues.apache.org/jira/browse/HIVE-24135
> Project: Hive
> Issue Type: Sub-task
>Reporter: Karen Coppage
>Assignee: Naveen Gangam
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Repro:
> say the default managed location is managed/hive and the default external
> location is external/hive.
> {code:java}
> create database db1; -- creates: external/hive/db1.db
> create table db1.table1 (i int); -- creates: managed/hive/db1.db and
> managed/hive/db1.db/table1
> drop database db1 cascade; -- removes : external/hive/db1.db and
> managed/hive/db1.db/table1
> {code}
> Problem: Directory managed/hive/db1.db remains.
> Since HIVE-22995, dbs have a managed (managedLocationUri) and an external
> location (locationUri). I think the issue is that
> HiveMetaStore.HMSHandler#drop_database_core deletes only the db directory in
> the external location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24155) Upgrade Arrow version to 1.0.1
[ https://issues.apache.org/jira/browse/HIVE-24155?focusedWorklogId=516413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516413 ] ASF GitHub Bot logged work on HIVE-24155: - Author: ASF GitHub Bot Created on: 25/Nov/20 00:43 Start Date: 25/Nov/20 00:43 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #1493: URL: https://github.com/apache/hive/pull/1493 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516413) Time Spent: 1h 10m (was: 1h) > Upgrade Arrow version to 1.0.1 > -- > > Key: HIVE-24155 > URL: https://issues.apache.org/jira/browse/HIVE-24155 > Project: Hive > Issue Type: Task >Affects Versions: 3.1.2 >Reporter: Igor Dvorzhak >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24155-branch-3.1.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24389) Trailing zeros of constant decimal numbers are removed
[
https://issues.apache.org/jira/browse/HIVE-24389?focusedWorklogId=516347=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516347
]
ASF GitHub Bot logged work on HIVE-24389:
-
Author: ASF GitHub Bot
Created on: 24/Nov/20 21:16
Start Date: 24/Nov/20 21:16
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1676:
URL: https://github.com/apache/hive/pull/1676#discussion_r529883008
##
File path:
ql/src/test/results/clientpositive/llap/materialized_view_rewrite_window.q.out
##
@@ -166,7 +166,7 @@ POSTHOOK: Input: arc_view@wealth
A masked pattern was here
CBO PLAN:
HiveSortLimit(sort0=[$0], dir0=[ASC])
- HiveProject(quartile=[$0], total=[$1])
+ HiveProject(quartile=[CAST($0):DECIMAL(12, 1)], total=[$1])
Review comment:
Why is this expected? It seems the decimal value is not a constant?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516347)
Time Spent: 1h 40m (was: 1.5h)
> Trailing zeros of constant decimal numbers are removed
> --
>
> Key: HIVE-24389
> URL: https://issues.apache.org/jira/browse/HIVE-24389
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> In some case Hive removes trailing zeros of constant decimal numbers
> {code}
> select cast(1.1 as decimal(22, 2))
> 1.1
> {code}
> In this case *WritableConstantHiveDecimalObjectInspector* is used and this
> object inspector takes it's wrapped HiveDecimal scale instead of the scale
> specified in the wrapped typeinfo:
> {code}
> this = {WritableConstantHiveDecimalObjectInspector@14415}
> value = {HiveDecimalWritable@14426} "1.1"
> typeInfo = {DecimalTypeInfo@14421} "decimal(22,2)"{code}
> However in case of an expression with an aggregate function
> *WritableHiveDecimalObjectInspector* is used
> {code}
> select cast(sum(1.1) as decimal(22, 2))
> 1.10
> {code}
> {code}
> o = {HiveDecimalWritable@16633} "1.1"
> oi = {WritableHiveDecimalObjectInspector@16634}
> typeInfo = {DecimalTypeInfo@16640} "decimal(22,2)"
> {code}
> Casting the expressions to string
> {code:java}
> select cast(cast(1.1 as decimal(22, 2)) as string), cast(cast(sum(1.1) as
> decimal(22, 2)) as string)
> 1.1 1.10
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21174) hive.stats.ndv.error parameter documentation issue
[ https://issues.apache.org/jira/browse/HIVE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-21174: - Fix Version/s: (was: 3.10) > hive.stats.ndv.error parameter documentation issue > -- > > Key: HIVE-21174 > URL: https://issues.apache.org/jira/browse/HIVE-21174 > Project: Hive > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.2.0, 2.3.0, 3.0.0, 2.3.1, > 2.3.2, 3.1.0, 2.0.2, 2.1.2, 2.2.1, 2.3.3, 2.3.4, 2.4.0, 3.10, 3.0.1, 3.1.1, > 3.1.2, 3.2.0 >Reporter: Pablo Junge >Assignee: Pablo Junge >Priority: Major > Fix For: 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.2.0, 2.3.0, 3.0.0, 2.3.1, > 2.3.2, 3.1.0, 2.0.2, 2.1.2, 2.2.1, 2.3.3, 2.3.4, 2.4.0, 3.0.1, 3.1.1, 3.2.0 > > > Hive documentation for hive.stats.ndv.error does not specify that > hive.stats.ndv.error will only affect FM Sketch and not HLL. > > https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24342) isPathEncrypted should make sure resolved path also from HDFS
[
https://issues.apache.org/jira/browse/HIVE-24342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HIVE-24342:
---
Description:
Currently isPathEncrypted will make sure path is from hdfs by checking the path
scheme is "hdfs"
In the case if mounted ViewFileSystem based files systems like
ViewFSOverloadScheme or ViewHDFS (HDFS-15289) may need o check resolved path is
really hdfs.
In ViewHDFS case, we can mount hdfs://ns1/test ---> o3fs://b.v.ozone1/test
When user calling queries with the path hdfs://ns1/test, isPathEncrypted will
think the path is from hdfs only as its checking path scheme.
{code:java}
0: jdbc:hive2://umag-1.umag.root.xxx.site:218> select * from test30;
Error: Error while compiling statement: FAILED: SemanticException Unable to
determine if hdfs://ns1/test is encrypted:
java.lang.UnsupportedOperationException: This API:getEZForPath is specific to
DFS. Can't run on other fs:o3fs://bucket.volume.ozone1 (state=42000,code=4)
0: jdbc:hive2://umag-1.umag.root.xxx.site:218> cd Closing: 0:
jdbc:hive2://umag-1.umag.root.xxx.site:2181,umag-2.umag.root.xxx.site:2181,umag-5.umag.root.xxx.site:2181/default;password=root;principal=hive/umag-5.umag.root.xxx.s...@root.hwx.site;retries=5;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
{code}
So, here we should use resolvePath to make sure the resolved path really in
hdfs. If the resolved path is not from hdfs (in above case, it o3fs path), then
it will return false.
After fixing this, the query is passing.:
{code:java}
0: jdbc:hive2://umag-1.umag.root.xxx.site:218> select * from test30;
INFO : Compiling
command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb):
select * from test30
INFO : No Stats for default@test30, Columns: item, user_id, state, order_id
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema:
Schema(fieldSchemas:[FieldSchema(name:test30.order_id, type:bigint,
comment:null), FieldSchema(name:test30.user_id, type:string, comment:null),
FieldSchema(name:test30.item, type:string, comment:null),
FieldSchema(name:test30.state, type:string, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb); Time
taken: 4.47 seconds
INFO : Executing
command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb):
select * from test30
INFO : Completed executing
command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb); Time
taken: 0.09 seconds
INFO : OK
+--+-+--+---+
| test30.order_id | test30.user_id | test30.item | test30.state |
+--+-+--+---+
| 1234 | u1 | iphone7 | CA |
| 2345 | u1 | ipad | CA |
| 3456 | u2 | desktop | NY |
+--+-+--+---+
11 rows selected (6.975 seconds)
{code}
was:
Currently isPathEncrypted will make sure path is from hdfs by check the path
scheme is "hdfs"
In the case if mounted ViewFileSystem based files systems like
ViewFSOverloadScheme or ViewHDFS (HDFS-15289) may need o check resolved path is
really hdfs.
In ViewHDFS case, we can mount hdfs://ns1/test ---> o3fs://b.v.ozone1/test
When user calling queries with the path hdfs://ns1/test, isPathEncrypted will
think the path is from hdfs only as its checking path scheme.
{code:java}
0: jdbc:hive2://umag-1.umag.root.xxx.site:218> select * from test30;
Error: Error while compiling statement: FAILED: SemanticException Unable to
determine if hdfs://ns1/test is encrypted:
java.lang.UnsupportedOperationException: This API:getEZForPath is specific to
DFS. Can't run on other fs:o3fs://bucket.volume.ozone1 (state=42000,code=4)
0: jdbc:hive2://umag-1.umag.root.xxx.site:218> cd Closing: 0:
jdbc:hive2://umag-1.umag.root.xxx.site:2181,umag-2.umag.root.xxx.site:2181,umag-5.umag.root.xxx.site:2181/default;password=root;principal=hive/umag-5.umag.root.xxx.s...@root.hwx.site;retries=5;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
{code}
So, here we should use resolvePath to make sure the resolved path really in
hdfs. If the resolved path is not from hdfs (in above case, it o3fs path), then
it will return false.
After fixing this, the query is passing.:
{code:java}
0: jdbc:hive2://umag-1.umag.root.xxx.site:218> select * from test30;
INFO : Compiling
command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb):
select * from test30
INFO : No Stats for default@test30, Columns: item, user_id, state, order_id
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema:
[jira] [Work logged] (HIVE-24424) Use PreparedStatements in DbNotificationListener getNextNLId
[ https://issues.apache.org/jira/browse/HIVE-24424?focusedWorklogId=516302=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516302 ] ASF GitHub Bot logged work on HIVE-24424: - Author: ASF GitHub Bot Created on: 24/Nov/20 18:42 Start Date: 24/Nov/20 18:42 Worklog Time Spent: 10m Work Description: belugabehr opened a new pull request #1704: URL: https://github.com/apache/hive/pull/1704 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516302) Remaining Estimate: 0h Time Spent: 10m > Use PreparedStatements in DbNotificationListener getNextNLId > > > Key: HIVE-24424 > URL: https://issues.apache.org/jira/browse/HIVE-24424 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Simplify the code, remove debug logging concatenation, and make it more > readable, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24424) Use PreparedStatements in DbNotificationListener getNextNLId
[ https://issues.apache.org/jira/browse/HIVE-24424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24424: -- Labels: pull-request-available (was: ) > Use PreparedStatements in DbNotificationListener getNextNLId > > > Key: HIVE-24424 > URL: https://issues.apache.org/jira/browse/HIVE-24424 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Simplify the code, remove debug logging concatenation, and make it more > readable, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-24396) [New Feature] Add data connector support for remote datasources
[ https://issues.apache.org/jira/browse/HIVE-24396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-24396 started by Naveen Gangam. > [New Feature] Add data connector support for remote datasources > --- > > Key: HIVE-24396 > URL: https://issues.apache.org/jira/browse/HIVE-24396 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > > This feature work is to be able to support in Hive Metastore to be able to > configure data connectors for remote datasources and map databases. We > currently have support for remote tables via StorageHandlers like > JDBCStorageHandler and HBaseStorageHandler. > Data connectors are a natural extension to this where we can map an entire > database or catalogs instead of individual tables. The tables within are > automagically mapped at runtime. The metadata for these tables are not > persisted in Hive. They are always mapped and built at runtime. > With this feature, we introduce a concept of type for Databases in Hive. > NATIVE vs REMOTE. All current databases are NATIVE. To create a REMOTE > database, the following syntax is to be used > CREATE REMOTE DATABASE remote_db USING WITH DCPROPERTIES > (); > Will attach a design doc to this jira. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24425) Create table in REMOTE db should fail
[ https://issues.apache.org/jira/browse/HIVE-24425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam reassigned HIVE-24425: > Create table in REMOTE db should fail > - > > Key: HIVE-24425 > URL: https://issues.apache.org/jira/browse/HIVE-24425 > Project: Hive > Issue Type: Sub-task >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > > Currently it creates the table in that DB but show tables does not show > anything. Preventing the creation of table will resolve this inconsistency > too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24408) Upgrade Parquet to 1.11.1
[ https://issues.apache.org/jira/browse/HIVE-24408?focusedWorklogId=516301=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516301 ] ASF GitHub Bot logged work on HIVE-24408: - Author: ASF GitHub Bot Created on: 24/Nov/20 18:38 Start Date: 24/Nov/20 18:38 Worklog Time Spent: 10m Work Description: jcamachor merged pull request #1692: URL: https://github.com/apache/hive/pull/1692 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516301) Time Spent: 1h 20m (was: 1h 10m) > Upgrade Parquet to 1.11.1 > - > > Key: HIVE-24408 > URL: https://issues.apache.org/jira/browse/HIVE-24408 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Parquet 1.11.1 has some bug fixes so Hive should consider to upgrade to it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24408) Upgrade Parquet to 1.11.1
[ https://issues.apache.org/jira/browse/HIVE-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-24408. Fix Version/s: 4.0.0 Resolution: Fixed Pushed to master, thanks for your contribution [~csun]! > Upgrade Parquet to 1.11.1 > - > > Key: HIVE-24408 > URL: https://issues.apache.org/jira/browse/HIVE-24408 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Parquet 1.11.1 has some bug fixes so Hive should consider to upgrade to it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24424) Use PreparedStatements in DbNotificationListener getNextNLId
[ https://issues.apache.org/jira/browse/HIVE-24424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-24424: - > Use PreparedStatements in DbNotificationListener getNextNLId > > > Key: HIVE-24424 > URL: https://issues.apache.org/jira/browse/HIVE-24424 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > > Simplify the code, remove debug logging concatenation, and make it more > readable, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-12587) Support to add partitioned data set to TestPerfCliDriver
[ https://issues.apache.org/jira/browse/HIVE-12587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-12587. Fix Version/s: 4.0.0 Assignee: Stamatis Zampetakis Resolution: Fixed > Support to add partitioned data set to TestPerfCliDriver > > > Key: HIVE-12587 > URL: https://issues.apache.org/jira/browse/HIVE-12587 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Stamatis Zampetakis >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HIVE-12587) Support to add partitioned data set to TestPerfCliDriver
[ https://issues.apache.org/jira/browse/HIVE-12587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reopened HIVE-12587: > Support to add partitioned data set to TestPerfCliDriver > > > Key: HIVE-12587 > URL: https://issues.apache.org/jira/browse/HIVE-12587 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-12587) Support to add partitioned data set to TestPerfCliDriver
[ https://issues.apache.org/jira/browse/HIVE-12587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-12587. Resolution: Duplicate > Support to add partitioned data set to TestPerfCliDriver > > > Key: HIVE-12587 > URL: https://issues.apache.org/jira/browse/HIVE-12587 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-23965) Improve plan regression tests using TPCDS30TB metastore dump and custom configs
[ https://issues.apache.org/jira/browse/HIVE-23965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-23965. Fix Version/s: 4.0.0 Resolution: Fixed Pushed to master, thanks [~zabetak]! > Improve plan regression tests using TPCDS30TB metastore dump and custom > configs > --- > > Key: HIVE-23965 > URL: https://issues.apache.org/jira/browse/HIVE-23965 > Project: Hive > Issue Type: Improvement >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > The existing regression tests (HIVE-12586) based on TPC-DS have certain > shortcomings: > The table statistics do not reflect cardinalities from a specific TPC-DS > scale factor (SF). Some tables are from a 30TB dataset, others from 200GB > dataset, and others from a 3GB dataset. This mix leads to plans that may > never appear when using an actual TPC-DS dataset. > The existing statistics do not contain information about partitions something > that can have a big impact on the resulting plans. > The existing regression tests rely on more or less on the default > configuration (hive-site.xml). In real-life scenarios though some of the > configurations differ and may impact the choices of the optimizer. > This issue aims to address the above shortcomings by using a curated > TPCDS30TB metastore dump along with some custom hive configurations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-23742) Remove unintentional execution of TPC-DS query39 in qtests
[ https://issues.apache.org/jira/browse/HIVE-23742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-23742. Fix Version/s: 4.0.0 Resolution: Fixed Pushed to master, thanks [~zabetak]! > Remove unintentional execution of TPC-DS query39 in qtests > -- > > Key: HIVE-23742 > URL: https://issues.apache.org/jira/browse/HIVE-23742 > Project: Hive > Issue Type: Task >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > TPC-DS queries under clientpositive/perf are meant only to check plan > regressions so they should never be really executed thus the execution part > should be removed from query39.q and cbo_query39.q -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24395) Intermittent failures to initialize dockerized Postgres metastore in tests
[
https://issues.apache.org/jira/browse/HIVE-24395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez resolved HIVE-24395.
Resolution: Fixed
Pushed to master, thanks [~zabetak]!
> Intermittent failures to initialize dockerized Postgres metastore in tests
> --
>
> Key: HIVE-24395
> URL: https://issues.apache.org/jira/browse/HIVE-24395
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Minor
> Fix For: 4.0.0
>
>
> In some cases tests relying on a dockerized Postgres metastore (see [Postgres
> JUnit
> rule|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/Postgres.java])
> fail to establish a connection with the database; as a consequence the
> HikariPool cannot be created and after multiple failed attempts the test
> fails.
> The following exception appears in the logs indicating a problem with reading
> from the socket.
> {noformat}
> 2020-11-16T15:52:03,075 DEBUG [main] pool.HikariPool: HikariPool-1 - Cannot
> acquire connection from data source
> org.postgresql.util.PSQLException: The connection attempt failed.
> at
> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:297)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
> ~[postgresql-42.2.14.jar:42.2.14]
> at org.postgresql.jdbc.PgConnection.(PgConnection.java:217)
> ~[postgresql-42.2.14.jar:42.2.14]
> at org.postgresql.Driver.makeConnection(Driver.java:458)
> ~[postgresql-42.2.14.jar:42.2.14]
> at org.postgresql.Driver.connect(Driver.java:260)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
> ~[HikariCP-2.6.1.jar:?]
> at
> com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
> ~[HikariCP-2.6.1.jar:?]
> at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356)
> ~[HikariCP-2.6.1.jar:?]
> at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199)
> ~[HikariCP-2.6.1.jar:?]
> at
> com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444)
> ~[HikariCP-2.6.1.jar:?]
> at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515)
> ~[HikariCP-2.6.1.jar:?]
> at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:112)
> ~[HikariCP-2.6.1.jar:?]
> at com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:72)
> ~[HikariCP-2.6.1.jar:?]
> at
> org.apache.hadoop.hive.metastore.datasource.HikariCPDataSourceProvider.create(HikariCPDataSourceProvider.java:87)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.PersistenceManagerProvider.initPMF(PersistenceManagerProvider.java:235)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.PersistenceManagerProvider.lambda$updatePmfProperties$0(PersistenceManagerProvider.java:212)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.PersistenceManagerProvider.retry(PersistenceManagerProvider.java:521)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(PersistenceManagerProvider.java:212)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:334)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:59)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:866)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:834)
>
[jira] [Work logged] (HIVE-23965) Improve plan regression tests using TPCDS30TB metastore dump and custom configs
[ https://issues.apache.org/jira/browse/HIVE-23965?focusedWorklogId=516298=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516298 ] ASF GitHub Bot logged work on HIVE-23965: - Author: ASF GitHub Bot Created on: 24/Nov/20 18:30 Start Date: 24/Nov/20 18:30 Worklog Time Spent: 10m Work Description: asfgit closed pull request #1347: URL: https://github.com/apache/hive/pull/1347 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516298) Time Spent: 5h 20m (was: 5h 10m) > Improve plan regression tests using TPCDS30TB metastore dump and custom > configs > --- > > Key: HIVE-23965 > URL: https://issues.apache.org/jira/browse/HIVE-23965 > Project: Hive > Issue Type: Improvement >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Time Spent: 5h 20m > Remaining Estimate: 0h > > The existing regression tests (HIVE-12586) based on TPC-DS have certain > shortcomings: > The table statistics do not reflect cardinalities from a specific TPC-DS > scale factor (SF). Some tables are from a 30TB dataset, others from 200GB > dataset, and others from a 3GB dataset. This mix leads to plans that may > never appear when using an actual TPC-DS dataset. > The existing statistics do not contain information about partitions something > that can have a big impact on the resulting plans. > The existing regression tests rely on more or less on the default > configuration (hive-site.xml). In real-life scenarios though some of the > configurations differ and may impact the choices of the optimizer. > This issue aims to address the above shortcomings by using a curated > TPCDS30TB metastore dump along with some custom hive configurations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23553) Bump ORC version to 1.6
[ https://issues.apache.org/jira/browse/HIVE-23553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23553: -- Summary: Bump ORC version to 1.6 (was: Bump ORC version) > Bump ORC version to 1.6 > --- > > Key: HIVE-23553 > URL: https://issues.apache.org/jira/browse/HIVE-23553 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > > Make sure we are using one of the more recent orc.versions that include > row-filtering (ORC-577 , ORC-622) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23553) Bump ORC version to 1.6
[ https://issues.apache.org/jira/browse/HIVE-23553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23553: -- Description: was:Make sure we are using one of the more recent orc.versions that include row-filtering (ORC-577 , ORC-622) > Bump ORC version to 1.6 > --- > > Key: HIVE-23553 > URL: https://issues.apache.org/jira/browse/HIVE-23553 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24414) Backport HIVE-19662 to branch-3.1
[ https://issues.apache.org/jira/browse/HIVE-24414?focusedWorklogId=516264=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516264 ] ASF GitHub Bot logged work on HIVE-24414: - Author: ASF GitHub Bot Created on: 24/Nov/20 17:38 Start Date: 24/Nov/20 17:38 Worklog Time Spent: 10m Work Description: sunchao commented on pull request #1698: URL: https://github.com/apache/hive/pull/1698#issuecomment-733131515 There are no new failures from this PR - all failures are existing. @aihuaxu could you help to review this? thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516264) Time Spent: 20m (was: 10m) > Backport HIVE-19662 to branch-3.1 > - > > Key: HIVE-24414 > URL: https://issues.apache.org/jira/browse/HIVE-24414 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > This JIRA proposes to backport HIVE-19662 to branch-3.1 and upgrade Avro to > 1.8.2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-24324) Remove deprecated API usage from Avro
[
https://issues.apache.org/jira/browse/HIVE-24324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238282#comment-17238282
]
Chao Sun edited comment on HIVE-24324 at 11/24/20, 5:37 PM:
[~pgaref], [~aihuaxu] in order to backport this to 3.1 we'll first need to
backport HIVE-19662 to 3.1. I've opened [this
PR|https://github.com/apache/hive/pull/1698]. Could you review it? Thanks.
was (Author: csun):
[~pgaref] in order to backport this to 3.1 we'll first need to backport
HIVE-19662 to 3.1. I've opened [this
PR|https://github.com/apache/hive/pull/1698]. Could you review it? Thanks.
> Remove deprecated API usage from Avro
> -
>
> Key: HIVE-24324
> URL: https://issues.apache.org/jira/browse/HIVE-24324
> Project: Hive
> Issue Type: Improvement
> Components: Avro
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {{JsonProperties#getJsonProp}} has been marked as deprecated in Avro 1.8 and
> removed since Avro 1.9. This replaces the API usage for this with
> {{getObjectProp}} which doesn't leak Json node from jackson. This will help
> downstream apps to depend on Hive while using higher version of Avro, and
> also help Hive to upgrade Avro version itself.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24324) Remove deprecated API usage from Avro
[
https://issues.apache.org/jira/browse/HIVE-24324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238282#comment-17238282
]
Chao Sun commented on HIVE-24324:
-
[~pgaref] in order to backport this to 3.1 we'll first need to backport
HIVE-19662 to 3.1. I've opened [this
PR|https://github.com/apache/hive/pull/1698]. Could you review it? Thanks.
> Remove deprecated API usage from Avro
> -
>
> Key: HIVE-24324
> URL: https://issues.apache.org/jira/browse/HIVE-24324
> Project: Hive
> Issue Type: Improvement
> Components: Avro
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {{JsonProperties#getJsonProp}} has been marked as deprecated in Avro 1.8 and
> removed since Avro 1.9. This replaces the API usage for this with
> {{getObjectProp}} which doesn't leak Json node from jackson. This will help
> downstream apps to depend on Hive while using higher version of Avro, and
> also help Hive to upgrade Avro version itself.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24051) Hive lineage information exposed in ExecuteWithHookContext
[ https://issues.apache.org/jira/browse/HIVE-24051?focusedWorklogId=516261=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516261 ] ASF GitHub Bot logged work on HIVE-24051: - Author: ASF GitHub Bot Created on: 24/Nov/20 17:35 Start Date: 24/Nov/20 17:35 Worklog Time Spent: 10m Work Description: sunchao commented on pull request #1413: URL: https://github.com/apache/hive/pull/1413#issuecomment-733129475 Oops. I forgot about this. Already approved. I'll merge it once the CI finishes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516261) Time Spent: 1h 20m (was: 1h 10m) > Hive lineage information exposed in ExecuteWithHookContext > -- > > Key: HIVE-24051 > URL: https://issues.apache.org/jira/browse/HIVE-24051 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24051.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The lineage information is not populated unless certain hooks are enabled. > However, this is a bit fragile, and no way for another hook that we write to > get this information. This proposes a flag to enable this instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24051) Hive lineage information exposed in ExecuteWithHookContext
[ https://issues.apache.org/jira/browse/HIVE-24051?focusedWorklogId=516260=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516260 ] ASF GitHub Bot logged work on HIVE-24051: - Author: ASF GitHub Bot Created on: 24/Nov/20 17:34 Start Date: 24/Nov/20 17:34 Worklog Time Spent: 10m Work Description: szehon-ho opened a new pull request #1413: URL: https://github.com/apache/hive/pull/1413 The lineage information is not populated unless certain hooks are enabled. However, this is a bit fragile, and no way for another hook that we write to get this information. This proposes a flag to enable this instead. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516260) Time Spent: 1h 10m (was: 1h) > Hive lineage information exposed in ExecuteWithHookContext > -- > > Key: HIVE-24051 > URL: https://issues.apache.org/jira/browse/HIVE-24051 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24051.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The lineage information is not populated unless certain hooks are enabled. > However, this is a bit fragile, and no way for another hook that we write to > get this information. This proposes a flag to enable this instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24423) Improve DbNotificationListener Thread
[
https://issues.apache.org/jira/browse/HIVE-24423?focusedWorklogId=516238=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516238
]
ASF GitHub Bot logged work on HIVE-24423:
-
Author: ASF GitHub Bot
Created on: 24/Nov/20 16:39
Start Date: 24/Nov/20 16:39
Worklog Time Spent: 10m
Work Description: belugabehr opened a new pull request #1703:
URL: https://github.com/apache/hive/pull/1703
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516238)
Remaining Estimate: 0h
Time Spent: 10m
> Improve DbNotificationListener Thread
> -
>
> Key: HIVE-24423
> URL: https://issues.apache.org/jira/browse/HIVE-24423
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Clean up and simplify {{DbNotificationListener}} thread class.
> Most importantly, stop the thread and wait for it to finish before launching
> a new thread.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24423) Improve DbNotificationListener Thread
[
https://issues.apache.org/jira/browse/HIVE-24423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24423:
--
Labels: pull-request-available (was: )
> Improve DbNotificationListener Thread
> -
>
> Key: HIVE-24423
> URL: https://issues.apache.org/jira/browse/HIVE-24423
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Clean up and simplify {{DbNotificationListener}} thread class.
> Most importantly, stop the thread and wait for it to finish before launching
> a new thread.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24423) Improve DbNotificationListener Thread
[
https://issues.apache.org/jira/browse/HIVE-24423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-24423:
-
> Improve DbNotificationListener Thread
> -
>
> Key: HIVE-24423
> URL: https://issues.apache.org/jira/browse/HIVE-24423
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> Clean up and simplify {{DbNotificationListener}} thread class.
> Most importantly, stop the thread and wait for it to finish before launching
> a new thread.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?focusedWorklogId=516202=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516202
]
ASF GitHub Bot logged work on HIVE-19253:
-
Author: ASF GitHub Bot
Created on: 24/Nov/20 15:34
Start Date: 24/Nov/20 15:34
Worklog Time Spent: 10m
Work Description: szehon-ho edited a comment on pull request #1537:
URL: https://github.com/apache/hive/pull/1537#issuecomment-733051181
Sorry I've been awhile. Thanks Naveen for taking a look. I had some free
time today to take a look and the TestHiveMetstoreTransformer but am still a
bit lost. I tried to set the table type to ManagedTable as you suggest, but
the MetastoreDefaultTransformer actually transforms it back to External table
by the time the assert happens (actually this should probably do it not via the
properties but by the modeled TableType, but in this case it doesn't matter).
Code:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L596.
So then after that , it run the asserts which fail as they seem to be for
ManagedTable, unless I am mistaken. If you have some time to let me know
anything else to try, would appreciate it. I haven't taken a look at Miguel's
comments yet.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516202)
Time Spent: 1h 50m (was: 1h 40m)
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alex Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: newbie, pull-request-available
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch, HIVE-19253.03.patch, HIVE-19253.04.patch,
> HIVE-19253.05.patch, HIVE-19253.06.patch, HIVE-19253.07.patch,
> HIVE-19253.08.patch, HIVE-19253.09.patch, HIVE-19253.10.patch,
> HIVE-19253.11.patch, HIVE-19253.12.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?focusedWorklogId=516203=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516203
]
ASF GitHub Bot logged work on HIVE-19253:
-
Author: ASF GitHub Bot
Created on: 24/Nov/20 15:34
Start Date: 24/Nov/20 15:34
Worklog Time Spent: 10m
Work Description: szehon-ho edited a comment on pull request #1537:
URL: https://github.com/apache/hive/pull/1537#issuecomment-733051181
Sorry I've been awhile. Thanks Naveen for taking a look. I had some free
time today to take a look and the TestHiveMetstoreTransformer but am still a
bit lost. I tried to set the table type to ManagedTable as you suggest, but
the MetastoreDefaultTransformer actually transforms it back to External table
by the time the assert happens (actually this should probably do it not via the
properties but by the modeled TableType, but in this case it doesn't matter).
Code:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L596.
So then after that , it run the asserts which fail as they seem to be testing
for ManagedTable, unless I am mistaken. If you have some time to let me know
anything else to try, would appreciate it. I haven't taken a look at Miguel's
comments yet.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516203)
Time Spent: 2h (was: 1h 50m)
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alex Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: newbie, pull-request-available
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch, HIVE-19253.03.patch, HIVE-19253.04.patch,
> HIVE-19253.05.patch, HIVE-19253.06.patch, HIVE-19253.07.patch,
> HIVE-19253.08.patch, HIVE-19253.09.patch, HIVE-19253.10.patch,
> HIVE-19253.11.patch, HIVE-19253.12.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?focusedWorklogId=516200=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516200
]
ASF GitHub Bot logged work on HIVE-19253:
-
Author: ASF GitHub Bot
Created on: 24/Nov/20 15:34
Start Date: 24/Nov/20 15:34
Worklog Time Spent: 10m
Work Description: szehon-ho edited a comment on pull request #1537:
URL: https://github.com/apache/hive/pull/1537#issuecomment-733051181
Sorry I've been awhile. Thanks Naveen for taking a look. I had some free
time today to take a look and the TestHiveMetstoreTransformer but am still a
bit lost. I tried to set the table type to ManagedTable as you suggest, but
the MetastoreDefaultTransformer actually sets transforms it back to External
table by the time the assert happens (actually this should probably do it not
via the properties but by the modeled TableType, but in this case it doesn't
matter). Code:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L596.
So then after that , it run the asserts which fail as they seem to be for
ManagedTable, unless I am mistaken. If you have some time to let me know
anything else to try, would appreciate it. I haven't taken a look at Miguel's
comments yet.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516200)
Time Spent: 1h 40m (was: 1.5h)
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alex Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: newbie, pull-request-available
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch, HIVE-19253.03.patch, HIVE-19253.04.patch,
> HIVE-19253.05.patch, HIVE-19253.06.patch, HIVE-19253.07.patch,
> HIVE-19253.08.patch, HIVE-19253.09.patch, HIVE-19253.10.patch,
> HIVE-19253.11.patch, HIVE-19253.12.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?focusedWorklogId=516199=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516199
]
ASF GitHub Bot logged work on HIVE-19253:
-
Author: ASF GitHub Bot
Created on: 24/Nov/20 15:33
Start Date: 24/Nov/20 15:33
Worklog Time Spent: 10m
Work Description: szehon-ho edited a comment on pull request #1537:
URL: https://github.com/apache/hive/pull/1537#issuecomment-733051181
Sorry I've been awhile. Thanks Naveen for taking a look. I had some free
time today to take a look and the TestHiveMetstoreTransformer but a, still a
bit lost. I tried to set the table type to ManagedTable as you suggest, but
the MetastoreDefaultTransformer actually sets transforms it back to External
table by the time the assert happens (actually this should probably do it not
via the properties but by the modeled TableType, but in this case it doesn't
matter). Code:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L596.
So then after that , it run the asserts which fail as they seem to be for
ManagedTable, unless I am mistaken. If you have some time to let me know
anything else to try, would appreciate it. I haven't taken a look at Miguel's
comments yet.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516199)
Time Spent: 1.5h (was: 1h 20m)
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alex Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: newbie, pull-request-available
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch, HIVE-19253.03.patch, HIVE-19253.04.patch,
> HIVE-19253.05.patch, HIVE-19253.06.patch, HIVE-19253.07.patch,
> HIVE-19253.08.patch, HIVE-19253.09.patch, HIVE-19253.10.patch,
> HIVE-19253.11.patch, HIVE-19253.12.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?focusedWorklogId=516197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516197
]
ASF GitHub Bot logged work on HIVE-19253:
-
Author: ASF GitHub Bot
Created on: 24/Nov/20 15:32
Start Date: 24/Nov/20 15:32
Worklog Time Spent: 10m
Work Description: szehon-ho commented on pull request #1537:
URL: https://github.com/apache/hive/pull/1537#issuecomment-733051181
Sorry I've been awhile. Thanks Naveen for taking a look. I had some free
time today to take a look and the TestHiveMetstoreTransformer but still a bit
lost. I tried to set the table type to ManagedTable, but the
MetastoreDefaultTransformer actually sets it back to External table by the time
the assert happens (actually this should probably do it not via the properties
but by the modeled TableType, but in this case it doesn't matter). Code:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L596.
So then after that , it will due the asserts which fail as they seem to be for
ManagedTable, unless I am mistaken. If you have some time to let me know
anything else to try, would appreciate it. I haven't taken a look at Miguel's
comments yet.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516197)
Time Spent: 1h 20m (was: 1h 10m)
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alex Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: newbie, pull-request-available
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch, HIVE-19253.03.patch, HIVE-19253.04.patch,
> HIVE-19253.05.patch, HIVE-19253.06.patch, HIVE-19253.07.patch,
> HIVE-19253.08.patch, HIVE-19253.09.patch, HIVE-19253.10.patch,
> HIVE-19253.11.patch, HIVE-19253.12.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24417) Add config options for Atlas and Ranger client timeouts
[ https://issues.apache.org/jira/browse/HIVE-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238178#comment-17238178 ] Aasha Medhi commented on HIVE-24417: +1 > Add config options for Atlas and Ranger client timeouts > --- > > Key: HIVE-24417 > URL: https://issues.apache.org/jira/browse/HIVE-24417 > Project: Hive > Issue Type: Task >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24417.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24422) Throw SemanticException when CTE alias is conflicted with table name
[
https://issues.apache.org/jira/browse/HIVE-24422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24422:
--
Labels: pull-request-available (was: )
> Throw SemanticException when CTE alias is conflicted with table name
>
>
> Key: HIVE-24422
> URL: https://issues.apache.org/jira/browse/HIVE-24422
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If the alias of CTE is conflicted with the table name, we use the alias
> fetching the table other than replacing it with the ASTNode tree, this may
> cause some confusing problems. For example:
> {noformat}
> create table game_info (game_name string);
> with game_info as (
> select distinct ext_id, dev_app_id, game_name
> from game_info_extend )
> select count(game_name) from game_info;{noformat}
> The query will return the number of rows of the table game_info, instead of
> the game_info_extend. Maybe we should better throw an exception to avoid such
> cases.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24422) Throw SemanticException when CTE alias is conflicted with table name
[
https://issues.apache.org/jira/browse/HIVE-24422?focusedWorklogId=516159=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516159
]
ASF GitHub Bot logged work on HIVE-24422:
-
Author: ASF GitHub Bot
Created on: 24/Nov/20 14:32
Start Date: 24/Nov/20 14:32
Worklog Time Spent: 10m
Work Description: dengzhhu653 opened a new pull request #1702:
URL: https://github.com/apache/hive/pull/1702
… table name
### What changes were proposed in this pull request?
Throw SemanticException when CTE alias is conflicted with table name
### Why are the changes needed?
If the alias of CTE is conflicted with the table name, we use the alias
fetching the table other than replacing it with the ASTNode tree, this may
cause some confusing problems. Throw exception instead to avoid such cases.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
unit test
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516159)
Remaining Estimate: 0h
Time Spent: 10m
> Throw SemanticException when CTE alias is conflicted with table name
>
>
> Key: HIVE-24422
> URL: https://issues.apache.org/jira/browse/HIVE-24422
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If the alias of CTE is conflicted with the table name, we use the alias
> fetching the table other than replacing it with the ASTNode tree, this may
> cause some confusing problems. For example:
> {noformat}
> create table game_info (game_name string);
> with game_info as (
> select distinct ext_id, dev_app_id, game_name
> from game_info_extend )
> select count(game_name) from game_info;{noformat}
> The query will return the number of rows of the table game_info, instead of
> the game_info_extend. Maybe we should better throw an exception to avoid such
> cases.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24422) Throw SemanticException when CTE alias is conflicted with table name
[
https://issues.apache.org/jira/browse/HIVE-24422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng reassigned HIVE-24422:
--
Assignee: Zhihua Deng
> Throw SemanticException when CTE alias is conflicted with table name
>
>
> Key: HIVE-24422
> URL: https://issues.apache.org/jira/browse/HIVE-24422
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
>
> If the alias of CTE is conflicted with the table name, we use the alias
> fetching the table other than replacing it with the ASTNode tree, this may
> cause some confusing problems. For example:
> {noformat}
> create table game_info (game_name string);
> with game_info as (
> select distinct ext_id, dev_app_id, game_name
> from game_info_extend )
> select count(game_name) from game_info;{noformat}
> The query will return the number of rows of the table game_info, instead of
> the game_info_extend. Maybe we should better throw an exception to avoid such
> cases.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-24420) Druid test failures
[ https://issues.apache.org/jira/browse/HIVE-24420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa resolved HIVE-24420. - Resolution: Fixed not an issue with latest apache, not seeing it there. was on a different branch > Druid test failures > > > Key: HIVE-24420 > URL: https://issues.apache.org/jira/browse/HIVE-24420 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > > Test Result (11 failures / ±0) > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz2] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_extractTime] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_floorTime] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_semijoin_reduction_all_types] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_ts] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24421) DruidOutputFormat and DruidStorageHandler use different filesystem causing issues in data loading
[ https://issues.apache.org/jira/browse/HIVE-24421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-24421: Description: Seeing below message in the logs - Directory /tmp/workingDir/.staging/segmentDir does not exist, ignore this if it is create statement or inserts of 0 rows, no Druid segments to move, cleaning working directory > DruidOutputFormat and DruidStorageHandler use different filesystem causing > issues in data loading > - > > Key: HIVE-24421 > URL: https://issues.apache.org/jira/browse/HIVE-24421 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > > Seeing below message in the logs - > Directory /tmp/workingDir/.staging/segmentDir does not exist, ignore this if > it is create statement or inserts of 0 rows, no Druid segments to move, > cleaning working directory -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24421) DruidOutputFormat and DruidStorageHandler use different filesystem causing issues in data loading
[ https://issues.apache.org/jira/browse/HIVE-24421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa reassigned HIVE-24421: --- > DruidOutputFormat and DruidStorageHandler use different filesystem causing > issues in data loading > - > > Key: HIVE-24421 > URL: https://issues.apache.org/jira/browse/HIVE-24421 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24420) Druid test failures
[ https://issues.apache.org/jira/browse/HIVE-24420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa reassigned HIVE-24420: --- > Druid test failures > > > Key: HIVE-24420 > URL: https://issues.apache.org/jira/browse/HIVE-24420 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > > Test Result (11 failures / ±0) > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz2] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_extractTime] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_floorTime] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_semijoin_reduction_all_types] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert] > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_ts] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24419) Refactor junit database rules to exploit testcontainers
[
https://issues.apache.org/jira/browse/HIVE-24419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis reassigned HIVE-24419:
--
> Refactor junit database rules to exploit testcontainers
> ---
>
> Key: HIVE-24419
> URL: https://issues.apache.org/jira/browse/HIVE-24419
> Project: Hive
> Issue Type: Task
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Minor
>
> The
> [DatabaseRule|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java]
> and its subclasses allow tests to run over dockerized metastores using
> different database backends. Essentially some part of the code manages
> containers by invoking explicitly docker commands.
> The [testcontainers|https://www.testcontainers.org/modules/databases/]
> project provides the necessary modules for managing dockerized databases in
> an easy and intuitive way.
> The goal of this issue is to refactor the {{DatabaseRule}} hierarchy to take
> advantage of testcontainers in order to delegate the burden of managing
> containers outside of Hive.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24417) Add config options for Atlas and Ranger client timeouts
[ https://issues.apache.org/jira/browse/HIVE-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha updated HIVE-24417: Attachment: HIVE-24417.01.patch > Add config options for Atlas and Ranger client timeouts > --- > > Key: HIVE-24417 > URL: https://issues.apache.org/jira/browse/HIVE-24417 > Project: Hive > Issue Type: Task >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24417.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24417) Add config options for Atlas and Ranger client timeouts
[ https://issues.apache.org/jira/browse/HIVE-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha updated HIVE-24417: Attachment: (was: HIVE-24417.01.patch) > Add config options for Atlas and Ranger client timeouts > --- > > Key: HIVE-24417 > URL: https://issues.apache.org/jira/browse/HIVE-24417 > Project: Hive > Issue Type: Task >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24417.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24417) Add config options for Atlas and Ranger client timeouts
[ https://issues.apache.org/jira/browse/HIVE-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha updated HIVE-24417: Summary: Add config options for Atlas and Ranger client timeouts (was: Add config options for Atlas client timeouts) > Add config options for Atlas and Ranger client timeouts > --- > > Key: HIVE-24417 > URL: https://issues.apache.org/jira/browse/HIVE-24417 > Project: Hive > Issue Type: Task >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24417.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24051) Hive lineage information exposed in ExecuteWithHookContext
[ https://issues.apache.org/jira/browse/HIVE-24051?focusedWorklogId=516021=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516021 ] ASF GitHub Bot logged work on HIVE-24051: - Author: ASF GitHub Bot Created on: 24/Nov/20 10:45 Start Date: 24/Nov/20 10:45 Worklog Time Spent: 10m Work Description: szehon-ho commented on pull request #1413: URL: https://github.com/apache/hive/pull/1413#issuecomment-732816511 Shoot sorry can you open this again when you have the chance? I still seem not to have permissions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516021) Time Spent: 1h (was: 50m) > Hive lineage information exposed in ExecuteWithHookContext > -- > > Key: HIVE-24051 > URL: https://issues.apache.org/jira/browse/HIVE-24051 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24051.patch > > Time Spent: 1h > Remaining Estimate: 0h > > The lineage information is not populated unless certain hooks are enabled. > However, this is a bit fragile, and no way for another hook that we write to > get this information. This proposes a flag to enable this instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24418) there is an error "java.lang.IllegalArgumentException: No columns to insert" when the result data is empty
[ https://issues.apache.org/jira/browse/HIVE-24418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HuiyuZhou updated HIVE-24418: - Description: i created the external hive table to link hbase, when i use hsql to insert data into hbase，there is an error "java.lang.IllegalArgumentException: No columns to insert", i search for the reason and found hbase client does not allow all empty column except rowkey to insert hbase. i also try to use "set hyperbase.fill.null.enable=true" to skip the error for my hsql, but it doest't work, how to avoid the error? is it a bug for this? was: i created the external hive table to link hbase, when i use hsql to insert data into hbase，there is an error "java.lang.IllegalArgumentException: No columns to insert", i search for the reason and found hbase client does not allow all empty column to insert hbase. i also try to use "set hyperbase.fill.null.enable=true" to skip the error for my hsql, but it doest't work, how to avoid the error? is it a bug for this? > there is an error "java.lang.IllegalArgumentException: No columns to insert" > when the result data is empty > -- > > Key: HIVE-24418 > URL: https://issues.apache.org/jira/browse/HIVE-24418 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 1.1.1 >Reporter: HuiyuZhou >Priority: Major > > i created the external hive table to link hbase, when i use hsql to insert > data into hbase，there is an error "java.lang.IllegalArgumentException: No > columns to insert", i search for the reason and found hbase client does not > allow all empty column except rowkey to insert hbase. > i also try to use "set hyperbase.fill.null.enable=true" to skip the error for > my hsql, but it doest't work, how to avoid the error? > is it a bug for this? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24415) HiveSplitGenerator blocks Tez dispatcher
[ https://issues.apache.org/jira/browse/HIVE-24415?focusedWorklogId=515967=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515967 ] ASF GitHub Bot logged work on HIVE-24415: - Author: ASF GitHub Bot Created on: 24/Nov/20 08:44 Start Date: 24/Nov/20 08:44 Worklog Time Spent: 10m Work Description: mustafaiman opened a new pull request #1701: URL: https://github.com/apache/hive/pull/1701 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 515967) Remaining Estimate: 0h Time Spent: 10m > HiveSplitGenerator blocks Tez dispatcher > > > Key: HIVE-24415 > URL: https://issues.apache.org/jira/browse/HIVE-24415 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa İman >Assignee: Mustafa İman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > HiveSplitGenerator does a lot of heavyweight operations in its constructor. > These operations block AsyncDispatcher in Tez > [https://github.com/apache/tez/blob/989d286d09cac7c4e4c5a0e06dd75ea5a6f15478/tez-dag/src/main/java/org/apache/tez/dag/app/dag/RootInputInitializerManager.java#L141] > . We should move bulk of initialization out of constructor. > The only reason of setting up everything in constructor is > DynamicPartitionPruner. We can buffer incoming events in HiveSplitGenerator > until dynamic partition pruner is initialized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24415) HiveSplitGenerator blocks Tez dispatcher
[ https://issues.apache.org/jira/browse/HIVE-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24415: -- Labels: pull-request-available (was: ) > HiveSplitGenerator blocks Tez dispatcher > > > Key: HIVE-24415 > URL: https://issues.apache.org/jira/browse/HIVE-24415 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa İman >Assignee: Mustafa İman >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HiveSplitGenerator does a lot of heavyweight operations in its constructor. > These operations block AsyncDispatcher in Tez > [https://github.com/apache/tez/blob/989d286d09cac7c4e4c5a0e06dd75ea5a6f15478/tez-dag/src/main/java/org/apache/tez/dag/app/dag/RootInputInitializerManager.java#L141] > . We should move bulk of initialization out of constructor. > The only reason of setting up everything in constructor is > DynamicPartitionPruner. We can buffer incoming events in HiveSplitGenerator > until dynamic partition pruner is initialized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24410) Query-based compaction hangs because of doAs
[
https://issues.apache.org/jira/browse/HIVE-24410?focusedWorklogId=515948=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515948
]
ASF GitHub Bot logged work on HIVE-24410:
-
Author: ASF GitHub Bot
Created on: 24/Nov/20 08:02
Start Date: 24/Nov/20 08:02
Worklog Time Spent: 10m
Work Description: pvargacl commented on a change in pull request #1693:
URL: https://github.com/apache/hive/pull/1693#discussion_r529272154
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##
@@ -590,6 +587,36 @@ public Object run() throws Exception {
return true;
}
+ private void failCompactionIfSetForTest() {
+if(conf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST) &&
conf.getBoolVar(HiveConf.ConfVars.HIVETESTMODEFAILCOMPACTION)) {
+ throw new
RuntimeException(HiveConf.ConfVars.HIVETESTMODEFAILCOMPACTION.name() + "=true");
+}
+ }
+
+ private void runCompactionViaMrJob(CompactionInfo ci, Table t, Partition p,
StorageDescriptor sd,
+ ValidCompactorWriteIdList tblValidWriteIds, StringBuilder jobName,
AcidUtils.Directory dir, StatsUpdater su)
+ throws IOException, HiveException, InterruptedException {
+final CompactorMR mr = new CompactorMR();
+if (runJobAsSelf(ci.runAs)) {
+ mr.run(conf, jobName.toString(), t, p, sd, tblValidWriteIds, ci, su,
msc, dir);
+} else {
+ UserGroupInformation ugi = UserGroupInformation.createProxyUser(ci.runAs,
+ UserGroupInformation.getLoginUser());
+ final Partition fp = p;
Review comment:
last nit: you don't need these final copies anymore, since they are
coming as parameters ad effectively finals.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515948)
Time Spent: 1h 40m (was: 1.5h)
> Query-based compaction hangs because of doAs
>
>
> Key: HIVE-24410
> URL: https://issues.apache.org/jira/browse/HIVE-24410
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> QB compaction runs within a doas +and+ hive.server2.enable.doAs is set to
> true (as of HIVE-24089). On a secure cluster with Worker threads running in
> HS2, this results in HMS client not receiving a login context during
> compaction queries, so kerberos prompts for a login via stdin which causes
> the worker thread to hang until it times out:
> {code:java}
> "node-x.com-44_executor" #47 daemon prio=1 os_prio=0 tid=0x01506000
> nid=0x1348 runnable [0x7f1beea95000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x9fa38c90> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at
> com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:862)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:708)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
[jira] [Updated] (HIVE-24417) Add config options for Atlas client timeouts
[ https://issues.apache.org/jira/browse/HIVE-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha updated HIVE-24417: Attachment: HIVE-24417.01.patch > Add config options for Atlas client timeouts > > > Key: HIVE-24417 > URL: https://issues.apache.org/jira/browse/HIVE-24417 > Project: Hive > Issue Type: Task >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24417.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24417) Add config options for Atlas client timeouts
[ https://issues.apache.org/jira/browse/HIVE-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha updated HIVE-24417: Status: Patch Available (was: Open) > Add config options for Atlas client timeouts > > > Key: HIVE-24417 > URL: https://issues.apache.org/jira/browse/HIVE-24417 > Project: Hive > Issue Type: Task >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24417.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
