[jira] [Assigned] (HIVE-27829) New command to display current connections on HS2 and HMS instances
[
https://issues.apache.org/jira/browse/HIVE-27829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi reassigned HIVE-27829:
---
Assignee: Riju Trivedi
> New command to display current connections on HS2 and HMS instances
> ---
>
> Key: HIVE-27829
> URL: https://issues.apache.org/jira/browse/HIVE-27829
> Project: Hive
> Issue Type: New Feature
> Components: Hive, HiveServer2, Standalone Metastore
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Riju Trivedi
>Priority: Major
>
> We would need a command to list current connections to HS2/HMS instances
> It could like {*}show processlist{*}(Mysql) or {*}select * from
> pg_stat_activity{*}(Postgresql) or {*}show compactions{*}(Hive) to see
> current connections to the Hive Server2/HMS instances
> This command can help in troubleshooting issues with Hive service. One can
> know the load on a given HS2/HMS instance with this command and identify
> inappropriate connections to terminate them.
>
> We can even extend this command to show connections between an HMS instance
> and backend database to troubleshoot issues between HMS and backend database
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-28075) Vectorized DayOFWeek returns inconsistent results for non-UTC timezones.
[
https://issues.apache.org/jira/browse/HIVE-28075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-28075 started by Riju Trivedi.
---
> Vectorized DayOFWeek returns inconsistent results for non-UTC timezones.
>
>
> Key: HIVE-28075
> URL: https://issues.apache.org/jira/browse/HIVE-28075
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-beta-1
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
>
> Simple problem reproduce -
> {code:java}
> --! qt:timezone:Asia/Shanghai
> CREATE EXTERNAL TABLE dayOfWeek_test(
> `fund_code` string,
> `test_date` string
> );
> INSERT INTO dayOfWeek_test(fund_code,test_date)
> values('SEC016210079','2023-04-13');
> SELECT fund_code,
> test_date,
> dayofweek(test_date) AS SR,
> CASE
> WHEN dayofweek(test_date) = 1 THEN 7
> ELSE dayofweek(test_date) - 1
> END AS week_day
> FROM dayOfWeek_test;
> Result :
> SEC0162100792023-04-13 4 3
> Expected Result:
> SEC016210079 2023-04-13 5 4
> {code}
> The issue is only with Vectorized path and non-UTC timezones. The
> non-vectorized path uses _DateTimeFormatter_ and the vectorized path __ uses
> _SimpleDateFormat_ and calendar initialized with UTC timezone. Hence, the
> local time zone date is converted to UTC which changes the date and
> dayOfWeek() result.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFDayOfWeekString.java#L59]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28075) Vectorized DayOFWeek returns inconsistent results for non-UTC timezones.
Riju Trivedi created HIVE-28075:
---
Summary: Vectorized DayOFWeek returns inconsistent results for
non-UTC timezones.
Key: HIVE-28075
URL: https://issues.apache.org/jira/browse/HIVE-28075
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 4.0.0-beta-1
Reporter: Riju Trivedi
Assignee: Riju Trivedi
Simple problem reproduce -
{code:java}
--! qt:timezone:Asia/Shanghai
CREATE EXTERNAL TABLE dayOfWeek_test(
`fund_code` string,
`test_date` string
);
INSERT INTO dayOfWeek_test(fund_code,test_date)
values('SEC016210079','2023-04-13');
SELECT fund_code,
test_date,
dayofweek(test_date) AS SR,
CASE
WHEN dayofweek(test_date) = 1 THEN 7
ELSE dayofweek(test_date) - 1
END AS week_day
FROM dayOfWeek_test;
Result :
SEC0162100792023-04-13 4 3
Expected Result:
SEC016210079 2023-04-13 5 4
{code}
The issue is only with Vectorized path and non-UTC timezones. The
non-vectorized path uses _DateTimeFormatter_ and the vectorized path __ uses
_SimpleDateFormat_ and calendar initialized with UTC timezone. Hence, the local
time zone date is converted to UTC which changes the date and dayOfWeek()
result.
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFDayOfWeekString.java#L59]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27923) Query fails with SemanticException if column referenced with name in ORDER by
[
https://issues.apache.org/jira/browse/HIVE-27923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27923:
Description:
Query fails with 'Invalid column reference' error if the column in Order by
clause is not referenced similar to Group By clause. Like below 2 queries will
compile correctly but the third one will fail.Atatched [^repro.q]
{code:java}
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`order_by_test`.`result_date`) DESC;
---Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
`process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Failure
{code}
Both column name and alias use should be supported for "ORDER BY" clause.
was:
Query fails with 'Invalid column reference' error if the column in Order by
clause is not referenced similar to Group By clause. Like below 2 queries will
compile correctly but the third one will fail.
{code:java}
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`order_by_test`.`result_date`) DESC;
---Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
`process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Failure
{code}
Both column name and alias use should be supported for "ORDER BY" clause.
Atatched [^repro.q]
> Query fails with SemanticException if column referenced with name in ORDER by
>
> ---
>
> Key: HIVE-27923
> URL: https://issues.apache.org/jira/browse/HIVE-27923
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: repro.q
>
>
> Query fails with 'Invalid column reference' error if the column in Order by
> clause is not referenced similar to Group By clause. Like below 2 queries
> will compile correctly but the third one will fail.Atatched [^repro.q]
> {code:java}
> CREATE VIEW view_order_by_test AS SELECT
> to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`order_by_test`.`result_date`)
> ORDER BY to_date(`order_by_test`.`result_date`) DESC;
> ---Successful
> CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
> `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`result_date`)
> ORDER BY to_date(`result_date`) DESC;
> --Successful
> CREATE VIEW view_order_by_test AS SELECT
> to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`order_by_test`.`result_date`)
> ORDER BY to_date(`result_date`) DESC;
> --Failure
> {code}
> Both column name and alias use should be supported for "ORDER BY" clause.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27923) Query fails with SemanticException if column referenced with name in ORDER by
[
https://issues.apache.org/jira/browse/HIVE-27923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27923:
Description:
Query fails with 'Invalid column reference' error if the column in Order by
clause is not referenced similar to Group By clause. Like below 2 queries will
compile correctly but the third one will fail.
{code:java}
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`order_by_test`.`result_date`) DESC;
---Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
`process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Failure
{code}
Both column name and alias use should be supported for "ORDER BY" clause.
Atatched [^repro.q]
was:
Query fails with 'Invalid column reference' error if the column in Order by
clause is not referenced similar to Group By clause. Like below 2 queries will
compile correctly but the third one will fail.
{code:java}
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`order_by_test`.`result_date`) DESC;
---Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
`process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Failure
{code}
Both column name and alias use should be supported for "ORDER BY" clause.
Attached repro.q
> Query fails with SemanticException if column referenced with name in ORDER by
>
> ---
>
> Key: HIVE-27923
> URL: https://issues.apache.org/jira/browse/HIVE-27923
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: repro.q
>
>
> Query fails with 'Invalid column reference' error if the column in Order by
> clause is not referenced similar to Group By clause. Like below 2 queries
> will compile correctly but the third one will fail.
> {code:java}
> CREATE VIEW view_order_by_test AS SELECT
> to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`order_by_test`.`result_date`)
> ORDER BY to_date(`order_by_test`.`result_date`) DESC;
> ---Successful
> CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
> `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`result_date`)
> ORDER BY to_date(`result_date`) DESC;
> --Successful
> CREATE VIEW view_order_by_test AS SELECT
> to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`order_by_test`.`result_date`)
> ORDER BY to_date(`result_date`) DESC;
> --Failure
> {code}
> Both column name and alias use should be supported for "ORDER BY" clause.
> Atatched [^repro.q]
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27923) Query fails with SemanticException if column referenced with name in ORDER by
[
https://issues.apache.org/jira/browse/HIVE-27923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27923:
Description:
Query fails with 'Invalid column reference' error if the column in Order by
clause is not referenced similar to Group By clause. Like below 2 queries will
compile correctly but the third one will fail. Attached [^repro.q]
{code:java}
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`order_by_test`.`result_date`) DESC;
---Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
`process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Failure
{code}
Both column name and alias use should be supported for the "ORDER BY" clause.
was:
Query fails with 'Invalid column reference' error if the column in Order by
clause is not referenced similar to Group By clause. Like below 2 queries will
compile correctly but the third one will fail.Atatched [^repro.q]
{code:java}
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`order_by_test`.`result_date`) DESC;
---Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
`process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Failure
{code}
Both column name and alias use should be supported for "ORDER BY" clause.
> Query fails with SemanticException if column referenced with name in ORDER by
>
> ---
>
> Key: HIVE-27923
> URL: https://issues.apache.org/jira/browse/HIVE-27923
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: repro.q
>
>
> Query fails with 'Invalid column reference' error if the column in Order by
> clause is not referenced similar to Group By clause. Like below 2 queries
> will compile correctly but the third one will fail. Attached [^repro.q]
> {code:java}
> CREATE VIEW view_order_by_test AS SELECT
> to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`order_by_test`.`result_date`)
> ORDER BY to_date(`order_by_test`.`result_date`) DESC;
> ---Successful
> CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
> `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`result_date`)
> ORDER BY to_date(`result_date`) DESC;
> --Successful
> CREATE VIEW view_order_by_test AS SELECT
> to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`order_by_test`.`result_date`)
> ORDER BY to_date(`result_date`) DESC;
> --Failure
> {code}
> Both column name and alias use should be supported for the "ORDER BY" clause.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27923) Query fails with SemanticException if column referenced with name in ORDER by
[
https://issues.apache.org/jira/browse/HIVE-27923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27923:
Attachment: repro.q
> Query fails with SemanticException if column referenced with name in ORDER by
>
> ---
>
> Key: HIVE-27923
> URL: https://issues.apache.org/jira/browse/HIVE-27923
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: repro.q
>
>
> Query fails with 'Invalid column reference' error if the column in Order by
> clause is not referenced similar to Group By clause. Like below 2 queries
> will compile correctly but the third one will fail.
> {code:java}
> CREATE VIEW view_order_by_test AS SELECT
> to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`order_by_test`.`result_date`)
> ORDER BY to_date(`order_by_test`.`result_date`) DESC;
> ---Successful
> CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
> `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`result_date`)
> ORDER BY to_date(`result_date`) DESC;
> --Successful
> CREATE VIEW view_order_by_test AS SELECT
> to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`order_by_test`.`result_date`)
> ORDER BY to_date(`result_date`) DESC;
> --Failure
> {code}
> Both column name and alias use should be supported for "ORDER BY" clause.
> Attached repro.q
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27923) Query fails with SemanticException if column referenced with name in ORDER by
[
https://issues.apache.org/jira/browse/HIVE-27923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27923:
Description:
Query fails with 'Invalid column reference' error if the column in Order by
clause is not referenced similar to Group By clause. Like below 2 queries will
compile correctly but the third one will fail.
{code:java}
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`order_by_test`.`result_date`) DESC;
---Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
`process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Failure
{code}
Both column name and alias use should be supported for "ORDER BY" clause.
Attached repro.q
was:
Query fails with 'Invalid column reference' error if the column in Order by
clause is not referenced similar to Group By clause. Like below 2 queries will
compile correctly but the third one will fail.
{code:java}
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`order_by_test`.`result_date`) DESC;
---Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
`process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Failure
{code}
Both column name and alias use should be supported for "ORDER BY" clause.
> Query fails with SemanticException if column referenced with name in ORDER by
>
> ---
>
> Key: HIVE-27923
> URL: https://issues.apache.org/jira/browse/HIVE-27923
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Riju Trivedi
>Priority: Major
>
> Query fails with 'Invalid column reference' error if the column in Order by
> clause is not referenced similar to Group By clause. Like below 2 queries
> will compile correctly but the third one will fail.
> {code:java}
> CREATE VIEW view_order_by_test AS SELECT
> to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`order_by_test`.`result_date`)
> ORDER BY to_date(`order_by_test`.`result_date`) DESC;
> ---Successful
> CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
> `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`result_date`)
> ORDER BY to_date(`result_date`) DESC;
> --Successful
> CREATE VIEW view_order_by_test AS SELECT
> to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`order_by_test`.`result_date`)
> ORDER BY to_date(`result_date`) DESC;
> --Failure
> {code}
> Both column name and alias use should be supported for "ORDER BY" clause.
> Attached repro.q
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27923) Query fails with SemanticException if column referenced with name in ORDER by
[
https://issues.apache.org/jira/browse/HIVE-27923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27923:
Description:
Query fails with 'Invalid column reference' error if the column in Order by
clause is not referenced similar to Group By clause. Like below 2 queries will
compile correctly but the third one will fail.
{code:java}
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`order_by_test`.`result_date`) DESC;
---Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
`process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Failure
{code}
Both column name and alias use should be supported for "ORDER BY" clause.
was:
Query fails with 'Invalid column reference' error if the column in Order by
clause is not referenced similar to Group By clause. Like below 2 queries will
compile correctly but the third one will fail.
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT(*) AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`order_by_test`.`result_date`) DESC;
---Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
`process_date`,COUNT(*) AS `count`
FROM `order_by_test`
GROUP BY to_date(`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Successful
CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`)
AS `process_date`,COUNT(*) AS `count`
FROM `order_by_test`
GROUP BY to_date(`order_by_test`.`result_date`)
ORDER BY to_date(`result_date`) DESC;
--Failure
> Query fails with SemanticException if column referenced with name in ORDER by
>
> ---
>
> Key: HIVE-27923
> URL: https://issues.apache.org/jira/browse/HIVE-27923
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Riju Trivedi
>Priority: Major
>
> Query fails with 'Invalid column reference' error if the column in Order by
> clause is not referenced similar to Group By clause. Like below 2 queries
> will compile correctly but the third one will fail.
> {code:java}
> CREATE VIEW view_order_by_test AS SELECT
> to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`order_by_test`.`result_date`)
> ORDER BY to_date(`order_by_test`.`result_date`) DESC;
> ---Successful
> CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS
> `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`result_date`)
> ORDER BY to_date(`result_date`) DESC;
> --Successful
> CREATE VIEW view_order_by_test AS SELECT
> to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT AS `count`
> FROM `order_by_test`
> GROUP BY to_date(`order_by_test`.`result_date`)
> ORDER BY to_date(`result_date`) DESC;
> --Failure
> {code}
> Both column name and alias use should be supported for "ORDER BY" clause.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27923) Query fails with SemanticException if column referenced with name in ORDER by
Riju Trivedi created HIVE-27923: --- Summary: Query fails with SemanticException if column referenced with name in ORDER by Key: HIVE-27923 URL: https://issues.apache.org/jira/browse/HIVE-27923 Project: Hive Issue Type: Bug Components: Logical Optimizer Reporter: Riju Trivedi Query fails with 'Invalid column reference' error if the column in Order by clause is not referenced similar to Group By clause. Like below 2 queries will compile correctly but the third one will fail. CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT(*) AS `count` FROM `order_by_test` GROUP BY to_date(`order_by_test`.`result_date`) ORDER BY to_date(`order_by_test`.`result_date`) DESC; ---Successful CREATE VIEW view_order_by_test AS SELECT to_date(`result_date`) AS `process_date`,COUNT(*) AS `count` FROM `order_by_test` GROUP BY to_date(`result_date`) ORDER BY to_date(`result_date`) DESC; --Successful CREATE VIEW view_order_by_test AS SELECT to_date(`order_by_test`.`result_date`) AS `process_date`,COUNT(*) AS `count` FROM `order_by_test` GROUP BY to_date(`order_by_test`.`result_date`) ORDER BY to_date(`result_date`) DESC; --Failure -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27788) Exception in Sort Merge join with Group By + PTF Operator
Riju Trivedi created HIVE-27788:
---
Summary: Exception in Sort Merge join with Group By + PTF Operator
Key: HIVE-27788
URL: https://issues.apache.org/jira/browse/HIVE-27788
Project: Hive
Issue Type: Bug
Components: Operators
Affects Versions: 4.0.0-beta-1
Reporter: Riju Trivedi
Attachments: auto_sortmerge_join_17.q
Sort- merge join with Group By + PTF operator leads to Runtime exception
{code:java}
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:313)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:293)
... 15 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:387)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:303)
... 17 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Attempting to overwrite
nextKeyWritables[1]
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:392)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:372)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:316)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:127)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.handleOutputRows(PTFOperator.java:337)
at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.processRow(PTFOperator.java:325)
at
org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:139)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:372)
... 18 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Attempting to overwrite
nextKeyWritables[1]
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:534)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:488)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:390)
... 31 more
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Attempting to overwrite
nextKeyWritables[1]
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:313)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:522)
... 33 more {code}
Issue can be reproduced with [^auto_sortmerge_join_17.q]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27113) Increasing default for hive.thrift.client.max.message.size to 2 GB
[
https://issues.apache.org/jira/browse/HIVE-27113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27113:
Description:
{code:java}
HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size",
"1gb",
new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
"Thrift client configuration for max message size. 0 or -1 will use the default
defined in the Thrift " +
"library. The upper limit is 2147483648 bytes (or 2gb).")
{code}
Documentation on the help suggests setting 2147483648 while Integer Max is
2147483647. So, it actually becomes -1 and gets set to thrift default limit
(100 MB)
was:
HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size",
"1gb",
new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
"Thrift client configuration for max message size. 0 or -1 will use the default
defined in the Thrift " +
"library. The upper limit is 2147483648 bytes (or 2gb).")
Documentation on the help suggests setting 2147483648 while Integer Max is
2147483647. So, it actually becomes -1 and gets set to thrift default limit
(100 MB)
> Increasing default for hive.thrift.client.max.message.size to 2 GB
> --
>
> Key: HIVE-27113
> URL: https://issues.apache.org/jira/browse/HIVE-27113
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
>
> {code:java}
> HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size",
> "1gb",
> new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
> "Thrift client configuration for max message size. 0 or -1 will use the
> default defined in the Thrift " +
> "library. The upper limit is 2147483648 bytes (or 2gb).")
>
> {code}
> Documentation on the help suggests setting 2147483648 while Integer Max is
> 2147483647. So, it actually becomes -1 and gets set to thrift default limit
> (100 MB)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27113) Increasing default for hive.thrift.client.max.message.size to 2 GB
[
https://issues.apache.org/jira/browse/HIVE-27113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27113:
Description:
HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size",
"1gb",
new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
"Thrift client configuration for max message size. 0 or -1 will use the default
defined in the Thrift " +
"library. The upper limit is 2147483648 bytes (or 2gb).")
Documentation on the help suggests setting 2147483648 while Integer Max is
2147483647. So, it actually becomes -1 and gets set to thrift default limit
(100 MB)
was:
HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size",
"1gb",
new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
"Thrift client configuration for max message size. 0 or -1 will use the
default defined in the Thrift " +
"library. The upper limit is 2147483648 bytes (or 2gb).")
Documentation on the help suggests setting 2147483648 while Integer Max is
2147483647. So, it actually becomes -1 and gets set to thrift default limit
(100 MB)
> Increasing default for hive.thrift.client.max.message.size to 2 GB
> --
>
> Key: HIVE-27113
> URL: https://issues.apache.org/jira/browse/HIVE-27113
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size",
> "1gb",
> new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
> "Thrift client configuration for max message size. 0 or -1 will use the
> default defined in the Thrift " +
> "library. The upper limit is 2147483648 bytes (or 2gb).")
> Documentation on the help suggests setting 2147483648 while Integer Max is
> 2147483647. So, it actually becomes -1 and gets set to thrift default limit
> (100 MB)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27636) Exception in HiveMaterializedViewsRegistry is leaving staging directories behind
[
https://issues.apache.org/jira/browse/HIVE-27636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27636:
Description:
In case of any exception while query parsing in
`HiveMaterializedViewsRegistry.createMaterialization`, we bail out and there is
no hdfs dir cleanup until JVM exit. This leaves behind the staging directories.
For a long-running HS2, these staging directories keeps on increasing and can
cause limit reached exception.
{code:java}
Error: Error while compiling statement: FAILED: RuntimeException Cannot create
staging directory
'hdfs://aidaprd01/warehouse/tablespace/managed/hive/test.db/testTable/.hive-staging_hive_2023-08-05_06-17-06_711_5516272990801215078-168329:
The directory item limit of
/warehouse/tablespace/managed/hive/test.db/testTable is exceeded: limit=1048576
items=1048576 {code}
We should do hdfs directory cleanup for `HiveMaterializedViewsRegistry` thread
[here|https://github.infra.cloudera.com/CDH/hive/blob/39b9e39e5167c8fcd35683f8e9e2c9a89fe86555/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java#L226]
was:
In case of any exception while query parsing in
`HiveMaterializedViewsRegistry.createMaterialization`, we bailout and there is
no hdfs dir cleanup until JVM exit. This leaves behind the staging directories.
For a long-running HS2, these staging directories keeps on increasing and can
cause limit reached exception.
{code:java}
Error: Error while compiling statement: FAILED: RuntimeException Cannot create
staging directory
'hdfs://aidaprd01/warehouse/tablespace/managed/hive/test.db/testTable/.hive-staging_hive_2023-08-05_06-17-06_711_5516272990801215078-168329:
The directory item limit of
/warehouse/tablespace/managed/hive/test.db/testTable is exceeded: limit=1048576
items=1048576 {code}
> Exception in HiveMaterializedViewsRegistry is leaving staging directories
> behind
>
>
> Key: HIVE-27636
> URL: https://issues.apache.org/jira/browse/HIVE-27636
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Reporter: Riju Trivedi
>Priority: Major
>
> In case of any exception while query parsing in
> `HiveMaterializedViewsRegistry.createMaterialization`, we bail out and there
> is no hdfs dir cleanup until JVM exit. This leaves behind the staging
> directories. For a long-running HS2, these staging directories keeps on
> increasing and can cause limit reached exception.
> {code:java}
> Error: Error while compiling statement: FAILED: RuntimeException Cannot
> create staging directory
> 'hdfs://aidaprd01/warehouse/tablespace/managed/hive/test.db/testTable/.hive-staging_hive_2023-08-05_06-17-06_711_5516272990801215078-168329:
> The directory item limit of
> /warehouse/tablespace/managed/hive/test.db/testTable is exceeded:
> limit=1048576 items=1048576 {code}
> We should do hdfs directory cleanup for `HiveMaterializedViewsRegistry`
> thread
> [here|https://github.infra.cloudera.com/CDH/hive/blob/39b9e39e5167c8fcd35683f8e9e2c9a89fe86555/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java#L226]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27636) Exception in HiveMaterializedViewsRegistry is leaving staging directories behind
[
https://issues.apache.org/jira/browse/HIVE-27636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27636:
Description:
In case of any exception while query parsing in
`HiveMaterializedViewsRegistry.createMaterialization`, we bailout and there is
no hdfs dir cleanup until JVM exit. This leaves behind the staging directories.
For a long-running HS2, these staging directories keeps on increasing and can
cause limit reached exception.
{code:java}
Error: Error while compiling statement: FAILED: RuntimeException Cannot create
staging directory
'hdfs://aidaprd01/warehouse/tablespace/managed/hive/test.db/testTable/.hive-staging_hive_2023-08-05_06-17-06_711_5516272990801215078-168329:
The directory item limit of
/warehouse/tablespace/managed/hive/test.db/testTable is exceeded: limit=1048576
items=1048576 {code}
> Exception in HiveMaterializedViewsRegistry is leaving staging directories
> behind
>
>
> Key: HIVE-27636
> URL: https://issues.apache.org/jira/browse/HIVE-27636
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Reporter: Riju Trivedi
>Priority: Major
>
> In case of any exception while query parsing in
> `HiveMaterializedViewsRegistry.createMaterialization`, we bailout and there
> is no hdfs dir cleanup until JVM exit. This leaves behind the staging
> directories. For a long-running HS2, these staging directories keeps on
> increasing and can cause limit reached exception.
> {code:java}
> Error: Error while compiling statement: FAILED: RuntimeException Cannot
> create staging directory
> 'hdfs://aidaprd01/warehouse/tablespace/managed/hive/test.db/testTable/.hive-staging_hive_2023-08-05_06-17-06_711_5516272990801215078-168329:
> The directory item limit of
> /warehouse/tablespace/managed/hive/test.db/testTable is exceeded:
> limit=1048576 items=1048576 {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27636) Exception in HiveMaterializedViewsRegistry is leaving staging directories behind
Riju Trivedi created HIVE-27636: --- Summary: Exception in HiveMaterializedViewsRegistry is leaving staging directories behind Key: HIVE-27636 URL: https://issues.apache.org/jira/browse/HIVE-27636 Project: Hive Issue Type: Bug Components: Materialized views Reporter: Riju Trivedi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27632) ClassCast Exception in Vectorization converting decimal64 to decimal
[
https://issues.apache.org/jira/browse/HIVE-27632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi reassigned HIVE-27632:
---
Assignee: Stephen Carlin
> ClassCast Exception in Vectorization converting decimal64 to decimal
>
>
> Key: HIVE-27632
> URL: https://issues.apache.org/jira/browse/HIVE-27632
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Riju Trivedi
>Assignee: Stephen Carlin
>Priority: Major
> Labels: pull-request-available
> Attachments: vectortest.q
>
>
> Attached [^vectortest.q] which fails with the below ClassCast Exception
> {code:java}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterDecimalColEqualDecimalScalar.evaluate(FilterDecimalColEqualDecimalScalar.java:64)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:919)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:171)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.deliverVectorizedRowBatch(VectorMapOperator.java:809)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:878)
> {code}
> This seems related to HIVE-26208 , which avoids Decimal64 to Decimal
> conversion for the vector expressions that explicitly handle decimal64 types.
> However, in this scenario exception comes from
> `FilterDecimalColEqualDecimalScalar`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27632) ClassCast Exception in Vectorization converting decimal64 to decimal
[
https://issues.apache.org/jira/browse/HIVE-27632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27632:
Attachment: vectortest.q
Component/s: Vectorization
Description:
Attached [^vectortest.q] which fails with the below ClassCast Exception
{code:java}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to
org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
at
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterDecimalColEqualDecimalScalar.evaluate(FilterDecimalColEqualDecimalScalar.java:64)
at
org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:125)
at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:919)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:171)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.deliverVectorizedRowBatch(VectorMapOperator.java:809)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:878)
{code}
This seems related to HIVE-26208 , which avoids Decimal64 to Decimal conversion
for the vector expressions that explicitly handle decimal64 types. However, in
this scenario exception comes from `FilterDecimalColEqualDecimalScalar`.
Summary: ClassCast Exception in Vectorization converting decimal64 to
decimal (was: ClassCast Exception in FilterDecimalColEqualDecimalScalar
converting decimal64 to decimal)
> ClassCast Exception in Vectorization converting decimal64 to decimal
>
>
> Key: HIVE-27632
> URL: https://issues.apache.org/jira/browse/HIVE-27632
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: vectortest.q
>
>
> Attached [^vectortest.q] which fails with the below ClassCast Exception
> {code:java}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterDecimalColEqualDecimalScalar.evaluate(FilterDecimalColEqualDecimalScalar.java:64)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:919)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:171)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.deliverVectorizedRowBatch(VectorMapOperator.java:809)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:878)
> {code}
> This seems related to HIVE-26208 , which avoids Decimal64 to Decimal
> conversion for the vector expressions that explicitly handle decimal64 types.
> However, in this scenario exception comes from
> `FilterDecimalColEqualDecimalScalar`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27632) ClassCast Exception in FilterDecimalColEqualDecimalScalar converting decimal64 to decimal
Riju Trivedi created HIVE-27632: --- Summary: ClassCast Exception in FilterDecimalColEqualDecimalScalar converting decimal64 to decimal Key: HIVE-27632 URL: https://issues.apache.org/jira/browse/HIVE-27632 Project: Hive Issue Type: Bug Reporter: Riju Trivedi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27629) Beeline should support set command after bracketed comments
[
https://issues.apache.org/jira/browse/HIVE-27629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27629:
Description:
Using beeline, hive fails to execute the below statement.
{noformat}
/* Bracketed coments */
set hive.variable.substitute=true;
select 1;
Error: Error while compiling statement: FAILED: ParseException line 2:4 cannot
recognize input near 'set' 'hive' '.' in statement
(state=42000,code=4){noformat}
SELECT/DDL/DML statements are supported after bracketed comments but
{color:#FF}set {color:#172b4d}does not work.{color}{color}
> Beeline should support set command after bracketed comments
> ---
>
> Key: HIVE-27629
> URL: https://issues.apache.org/jira/browse/HIVE-27629
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Riju Trivedi
>Priority: Minor
>
> Using beeline, hive fails to execute the below statement.
> {noformat}
> /* Bracketed coments */
> set hive.variable.substitute=true;
> select 1;
> Error: Error while compiling statement: FAILED: ParseException line 2:4
> cannot recognize input near 'set' 'hive' '.' in statement
> (state=42000,code=4){noformat}
> SELECT/DDL/DML statements are supported after bracketed comments but
> {color:#FF}set {color:#172b4d}does not work.{color}{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27629) Beeline should support set command after bracketed comments
Riju Trivedi created HIVE-27629: --- Summary: Beeline should support set command after bracketed comments Key: HIVE-27629 URL: https://issues.apache.org/jira/browse/HIVE-27629 Project: Hive Issue Type: Bug Components: Beeline Reporter: Riju Trivedi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27525) Ease the write permissions on external table during create table operation
[ https://issues.apache.org/jira/browse/HIVE-27525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-27525: Description: During the creation of external tables with a specified location, the general expectation is that the data is already present or the data might be externally added to the location without involving HMS. So, it is really not required to have read and write permissions on an external table during the creation time. This enhancement can address security concerns where currently the users had to be granted unnecessary write permissions on an external file location when the table is only used for reading the data. Update/delete operations would anyway require write permissions. CTAS query with location specified is expected to fail at runtime (trying to create staging directory under table location) without WRITE permissions. was: During the creation of external tables with a specified location, the general expectation is that the data is already present or the data might be externally added to the location without involving HMS. So, it is really not required to have read and write permissions on an external table during the creation time. This enhancement can address security concerns where currently the users had to be granted unnecessary write permissions on an external file location when the table is only used for reading the data. Update/delete operations would anyway require write permissions. CTAS query with location specified is expected to fail at runtime without , trying to create staging directory under table location. > Ease the write permissions on external table during create table operation > -- > > Key: HIVE-27525 > URL: https://issues.apache.org/jira/browse/HIVE-27525 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Riju Trivedi >Priority: Major > Labels: pull-request-available > > During the creation of external tables with a specified location, the general > expectation is that the data is already present or the data might be > externally added to the location without involving HMS. So, it is really not > required to have read and write permissions on an external table during the > creation time. > This enhancement can address security concerns where currently the users had > to be granted unnecessary write permissions on an external file location when > the table is only used for reading the data. > Update/delete operations would anyway require write permissions. > CTAS query with location specified is expected to fail at runtime (trying to > create staging directory under table location) without WRITE permissions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27525) Ease the write permissions on external table during create table operation
[ https://issues.apache.org/jira/browse/HIVE-27525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-27525: Description: During the creation of external tables with a specified location, the general expectation is that the data is already present or the data might be externally added to the location without involving HMS. So, it is really not required to have read and write permissions on an external table during the creation time. This enhancement can address security concerns where currently the users had to be granted unnecessary write permissions on an external file location when the table is only used for reading the data. Update/delete operations would anyway require write permissions. CTAS query with location specified is expected to fail at runtime without , trying to create staging directory under table location. was: During the creation of external tables with a specified location, the general expectation is that the data is already present or the data might be externally added to the location without involving HMS. So, it is really not required to have read and write permissions on an external table during the creation time. This enhancement can address security concerns where currently the users had to be granted unnecessary write permissions on an external file location when the table is only used for reading the data. Update/delete operations would anyway require write permissions. In case of CTAS with location specified, query would pass he authorization > Ease the write permissions on external table during create table operation > -- > > Key: HIVE-27525 > URL: https://issues.apache.org/jira/browse/HIVE-27525 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Riju Trivedi >Priority: Major > Labels: pull-request-available > > During the creation of external tables with a specified location, the general > expectation is that the data is already present or the data might be > externally added to the location without involving HMS. So, it is really not > required to have read and write permissions on an external table during the > creation time. > This enhancement can address security concerns where currently the users had > to be granted unnecessary write permissions on an external file location when > the table is only used for reading the data. > Update/delete operations would anyway require write permissions. > CTAS query with location specified is expected to fail at runtime without , > trying to create staging directory under table location. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27525) Ease the write permissions on external table during create table operation
[ https://issues.apache.org/jira/browse/HIVE-27525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-27525: Description: During the creation of external tables with a specified location, the general expectation is that the data is already present or the data might be externally added to the location without involving HMS. So, it is really not required to have read and write permissions on an external table during the creation time. This enhancement can address security concerns where currently the users had to be granted unnecessary write permissions on an external file location when the table is only used for reading the data. Update/delete operations would anyway require write permissions. In case of CTAS with location specified, query would pass he authorization was: During the creation of external tables with a specified location, the general expectation is that the data is already present or the data might be externally added to the location without involving HMS. So, it is really not required to have read and write permissions on an external table during the creation time. This enhancement can address security concerns where currently the users had to be granted unnecessary write permissions on an external file location when the table is only used for reading the data. Update/delete operations would anyway require write permissions. In case of CTAS with location specificed > Ease the write permissions on external table during create table operation > -- > > Key: HIVE-27525 > URL: https://issues.apache.org/jira/browse/HIVE-27525 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Riju Trivedi >Priority: Major > Labels: pull-request-available > > During the creation of external tables with a specified location, the general > expectation is that the data is already present or the data might be > externally added to the location without involving HMS. So, it is really not > required to have read and write permissions on an external table during the > creation time. > This enhancement can address security concerns where currently the users had > to be granted unnecessary write permissions on an external file location when > the table is only used for reading the data. > Update/delete operations would anyway require write permissions. > In case of CTAS with location specified, query would pass he authorization -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27525) Ease the write permissions on external table during create table operation
[ https://issues.apache.org/jira/browse/HIVE-27525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-27525: Description: During the creation of external tables with a specified location, the general expectation is that the data is already present or the data might be externally added to the location without involving HMS. So, it is really not required to have read and write permissions on an external table during the creation time. This enhancement can address security concerns where currently the users had to be granted unnecessary write permissions on an external file location when the table is only used for reading the data. Update/delete operations would anyway require write permissions. In case of CTAS with location specificed was: During the creation of external tables with a specified location, the general expectation is that the data is already present or the data might be externally added to the location without involving HMS. So, it is really not required to have read and write permissions on an external table during the creation time. This enhancement can address security concerns where currently the users had to be granted unnecessary write permissions on an external file location when the table is only used for reading the data. Update/delete operations would anyway require write permissions. > Ease the write permissions on external table during create table operation > -- > > Key: HIVE-27525 > URL: https://issues.apache.org/jira/browse/HIVE-27525 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Riju Trivedi >Priority: Major > Labels: pull-request-available > > During the creation of external tables with a specified location, the general > expectation is that the data is already present or the data might be > externally added to the location without involving HMS. So, it is really not > required to have read and write permissions on an external table during the > creation time. > This enhancement can address security concerns where currently the users had > to be granted unnecessary write permissions on an external file location when > the table is only used for reading the data. > Update/delete operations would anyway require write permissions. > In case of CTAS with location specificed -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27525) Ease the write permissions on external table during create table operation
[ https://issues.apache.org/jira/browse/HIVE-27525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-27525: Description: During the creation of external tables with a specified location, the general expectation is that the data is already present or the data might be externally added to the location without involving HMS. So, it is really not required to have read and write permissions on an external table during the creation time. This enhancement can address security concerns where currently the users had to be granted unnecessary write permissions on an external file location when the table is only used for reading the data. Update/delete operations would anyway require write permissions. was: During the creation of external tables with a specified location, the general expectation is that the data is already present or the data might be externally added to the location without involving HMS. So, it is really not required to have read and write permissions on an external table during the creation time. This enhancement can address security concerns where currently the users had to be grant unnecessary write permissions on an external file location when the table is only used for reading the data. Update/delete operations would anyway require write permissions. > Ease the write permissions on external table during create table operation > -- > > Key: HIVE-27525 > URL: https://issues.apache.org/jira/browse/HIVE-27525 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Riju Trivedi >Priority: Major > > During the creation of external tables with a specified location, the general > expectation is that the data is already present or the data might be > externally added to the location without involving HMS. So, it is really not > required to have read and write permissions on an external table during the > creation time. > This enhancement can address security concerns where currently the users had > to be granted unnecessary write permissions on an external file location when > the table is only used for reading the data. > Update/delete operations would anyway require write permissions. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27525) Ease the write permissions on external table during create table operation
[ https://issues.apache.org/jira/browse/HIVE-27525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi reassigned HIVE-27525: --- Assignee: Riju Trivedi (was: Sai Hemanth Gantasala) > Ease the write permissions on external table during create table operation > -- > > Key: HIVE-27525 > URL: https://issues.apache.org/jira/browse/HIVE-27525 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Riju Trivedi >Priority: Major > > During the creation of external tables with a specified location, the general > expectation is that the data is already present or the data might be > externally added to the location without involving HMS. So, it is really not > required to have read and write permissions on an external table during the > creation time. > This enhancement can address security concerns where currently the users had > to be grant unnecessary write permissions on an external file location when > the table is only used for reading the data. > Update/delete operations would anyway require write permissions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-22961) Drop function in Hive should not send request for drop database to Ranger plugin.
[
https://issues.apache.org/jira/browse/HIVE-22961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi reassigned HIVE-22961:
---
Assignee: Riju Trivedi (was: Sai Hemanth Gantasala)
> Drop function in Hive should not send request for drop database to Ranger
> plugin.
> --
>
> Key: HIVE-22961
> URL: https://issues.apache.org/jira/browse/HIVE-22961
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Riju Trivedi
>Priority: Major
>
> Issue here is how HIVE sends the "HivePrivilegeObjects" to Ranger when DROP
> fUNTION is done. This is different from how DROP TABLE is done.
> DROP TABLE the following is the request:
> {code:java}
> 'checkPrivileges':{'hiveOpType':DROPTABLE,
> 'inputHObjs':['HivePrivilegeObject':{'type':TABLE_OR_VIEW, 'dbName':testdemo,
> 'objectType':TABLE_OR_VIEW, 'objectName':t1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':systest}],
> 'outputHObjs':['HivePrivilegeObject':{'type':TABLE_OR_VIEW,
> 'dbName':testdemo, 'objectType':TABLE_OR_VIEW, 'objectName':t1, 'columns':[],
> 'partKeys':[], 'commandParams':[], 'actionType':OTHER, 'owner':systest}],
> 'context':{'clientType':HIVESERVER2, 'commandString':drop table t1,
> 'ipAddress':10.65.42.125, 'forwardedAddresses':null,
> 'sessionString':58f89a16-2df5-4124-af0e-913aabbefe06}, 'user':systest,
> 'groups':[systest, wheel]}{code}
> Where as in DROP FUNCTION:
> {code:java}
> {'hiveOpType':DROPFUNCTION,
> 'inputHObjs':['HivePrivilegeObject':{'type':FUNCTION, 'dbName':udfdemo,
> 'objectType':FUNCTION, 'objectName':aes1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':null}],
> 'outputHObjs':['HivePrivilegeObject':{'type':DATABASE, 'dbName':udfdemo,
> 'objectType':DATABASE, 'objectName':null, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER,
> 'owner':systest},'HivePrivilegeObject':{'type':FUNCTION, 'dbName':udfdemo,
> 'objectType':FUNCTION, 'objectName':aes1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':null}],
> 'context':{'clientType':HIVESERVER2, 'commandString':drop function
> udfdemo.aes1, 'ipAddress':10.65.42.125, 'forwardedAddresses':null,
> 'sessionString':442ca4d3-f34a-470c-878a-18542b99016c}, 'user':systest,
> 'groups':[systest, wheel]}
> {code}
> in DROP function in outputHObjs, there is this addition, DATABASE object
> which should not there and this causes the Ranger requested to be generated
> differently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746119#comment-17746119
]
Riju Trivedi edited comment on HIVE-27195 at 7/23/23 6:40 PM:
--
Thank you Stamatis for reviewing.
* These tests have default `hive.exec.drop.ignorenonexistent` to True hence
the behavior of DROP TABLE is the NOOP. I have added one more test with
`hive.exec.drop.ignorenonexistent` to False where DROP TABLE WITHOUT IF EXISTS
returns an error.
* Agreed, Updated tests to remove grant on tables.
* CREATE TABLE *IF NOT EXISTS* also throws an authentication error in case
table is already there. So, it is consistent with the DROP table IF EXISTS for
non-existing tables.
was (Author: rtrivedi12):
Thank you Stamatis for reviewing.
# These tests have default `hive.exec.drop.ignorenonexistent` to True hence
the behavior of DROP TABLE is the NOOP. I have added one more test with
`hive.exec.drop.ignorenonexistent` to False where DROP TABLE WITHOUT IF EXISTS
returns an error.
# Agreed, Updated tests to remove grant on tables.
# CREATE TABLE *IF NOT EXISTS* also throws an authentication error in case
table is already there.
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with the table object to the
> list of output objects sent for verifying privileges. This change would
> ensure that in case of a non-existent table or temporary table (skipped from
> authorization after HIVE-20051), the authorizer will verify privileges for
> the database object.
> This would also prevent DROP TABLE IF EXISTS command failure for temporary or
> non-existing tables with `RangerHiveAuthorizer`. In case of
> temporary/non-existing table, empty input and output HivePrivilege Objects
> are sent to Ranger authorizer and after
> https://issues.apache.org/jira/browse/RANGER-3407 authorization request is
> built from command in case of empty objects. Hence, the drop table if Exists
> command fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746119#comment-17746119
]
Riju Trivedi commented on HIVE-27195:
-
Thank you Stamatis for reviewing.
# These tests have default `hive.exec.drop.ignorenonexistent` to True hence
the behavior of DROP TABLE is the NOOP. I have added one more test with
`hive.exec.drop.ignorenonexistent` to False where DROP TABLE WITHOUT IF EXISTS
returns an error.
# Agreed, Updated tests to remove grant on tables.
# CREATE TABLE *IF NOT EXISTS* also throws an authentication error in case
table is already there.
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with the table object to the
> list of output objects sent for verifying privileges. This change would
> ensure that in case of a non-existent table or temporary table (skipped from
> authorization after HIVE-20051), the authorizer will verify privileges for
> the database object.
> This would also prevent DROP TABLE IF EXISTS command failure for temporary or
> non-existing tables with `RangerHiveAuthorizer`. In case of
> temporary/non-existing table, empty input and output HivePrivilege Objects
> are sent to Ranger authorizer and after
> https://issues.apache.org/jira/browse/RANGER-3407 authorization request is
> built from command in case of empty objects. Hence, the drop table if Exists
> command fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17745509#comment-17745509
]
Riju Trivedi commented on HIVE-27195:
-
Thank you [~zabetak] for reviewing and consolidating test scenarios. I have
updated the test results to the
[sheet|https://docs.google.com/spreadsheets/d/1CJ1U0LOCpK7TfxY5RSSM4Wmbmt7GiKt5VQrWt1x2tfs/edit?pli=1#gid=0]
and uploaded tests to the PR.
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with the table object to the
> list of output objects sent for verifying privileges. This change would
> ensure that in case of a non-existent table or temporary table (skipped from
> authorization after HIVE-20051), the authorizer will verify privileges for
> the database object.
> This would also prevent DROP TABLE IF EXISTS command failure for temporary or
> non-existing tables with `RangerHiveAuthorizer`. In case of
> temporary/non-existing table, empty input and output HivePrivilege Objects
> are sent to Ranger authorizer and after
> https://issues.apache.org/jira/browse/RANGER-3407 authorization request is
> built from command in case of empty objects. Hence, the drop table if Exists
> command fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27195:
Description:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), the authorizer will verify privileges for the
database object.
This would also prevent DROP TABLE IF EXISTS command failure for temporary or
non-existing tables with `RangerHiveAuthorizer`. In case of
temporary/non-existing table, empty input and output HivePrivilege Objects are
sent to Ranger authorizer and after
https://issues.apache.org/jira/browse/RANGER-3407 authorization request is
built from command in case of empty objects. Hence, the drop table if Exists
command fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
was:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), the authorizer will verify privileges for the
database object.
This would also prevent DROP TABLE IF EXISTS command failure for temporary or
non-existing tables with `RangerHiveAuthorizer`. In case of
temporary/non-existing table, empty input and output HivePrivilege Objects are
sent to Ranger authorizer and after
https://issues.apache.org/jira/browse/RANGER-3407 authorization request is
built from command in case of empty objects. Hence , the drop table if Exists
command fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with the table object to the
> list of output objects sent for verifying privileges. This change would
> ensure that in case of a non-existent table or temporary table (skipped from
> authorization after HIVE-20051), the authorizer will verify privileges for
> the database object.
> This would also prevent DROP TABLE IF EXISTS command failure for temporary or
> non-existing tables with `RangerHiveAuthorizer`. In case of
> temporary/non-existing table, empty input and output HivePrivilege Objects
> are sent to Ranger authorizer and after
> https://issues.apache.org/jira/browse/RANGER-3407 authorization request is
> built from command in case of empty objects. Hence, the drop table if Exists
> command fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27195:
Description:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), the authorizer will verify privileges for the
database object.
This would also prevent DROP TABLE IF EXISTS command failure for temporary or
non-existing tables with `RangerHiveAuthorizer`. In case of
temporary/non-existing table, empty input and output HivePrivilege Objects are
sent to Ranger authorizer and after
https://issues.apache.org/jira/browse/RANGER-3407 authorization request is
built from command in case of empty objects. Hence , the drop table if Exists
command fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
was:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), the authorizer will verify privileges for the
database object.
This would also prevent DROP TABLE IF EXISTS command failure for temporary or
non-existing tables with Ranger. In case of temporary/non-existing table, empty
input and output HivePrivilege Objects are sent to Ranger authorizer and after
https://issues.apache.org/jira/browse/RANGER-3407 authorization request is
build from command in case of empty objects. Hence
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with the table object to the
> list of output objects sent for verifying privileges. This change would
> ensure that in case of a non-existent table or temporary table (skipped from
> authorization after HIVE-20051), the authorizer will verify privileges for
> the database object.
> This would also prevent DROP TABLE IF EXISTS command failure for temporary or
> non-existing tables with `RangerHiveAuthorizer`. In case of
> temporary/non-existing table, empty input and output HivePrivilege Objects
> are sent to Ranger authorizer and after
> https://issues.apache.org/jira/browse/RANGER-3407 authorization request is
> built from command in case of empty objects. Hence , the drop table if Exists
> command fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27195:
Description:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), the authorizer will verify privileges for the
database object.
This would also prevent DROP TABLE IF EXISTS command failure for temporary or
non-existing tables with Ranger. In case of temporary/non-existing table, empty
input and output HivePrivilege Objects are sent to Ranger authorizer and after
https://issues.apache.org/jira/browse/RANGER-3407 authorization request is
build from command in case of empty objects. Hence
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
was:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), the authorizer will verify privileges for the
database object.
This would also prevent DROP TABLE IF EXISTS command failure for temporary or
non-existing tables with Ranger. In case of temporary/non-existing table, empty
input and output HivePrivilege Objects are sent to Ranger authroizer
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with the table object to the
> list of output objects sent for verifying privileges. This change would
> ensure that in case of a non-existent table or temporary table (skipped from
> authorization after HIVE-20051), the authorizer will verify privileges for
> the database object.
> This would also prevent DROP TABLE IF EXISTS command failure for temporary or
> non-existing tables with Ranger. In case of temporary/non-existing table,
> empty input and output HivePrivilege Objects are sent to Ranger authorizer
> and after https://issues.apache.org/jira/browse/RANGER-3407 authorization
> request is build from command in case of empty objects. Hence
> the drop table if Exists fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27195:
Description:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), the authorizer will verify privileges for the
database object.
This would also prevent DROP TABLE IF EXISTS command failure for temporary or
non-existing tables with Ranger. In case of temporary/non-existing table, empty
input and output HivePrivilege Objects are sent to Ranger authroizer
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
was:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), the authorizer will verify privileges for the
database object.
This would also prevent DROP TABLE IF EXISTS command failure for temporary or
non-existing table with Ranger
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with the table object to the
> list of output objects sent for verifying privileges. This change would
> ensure that in case of a non-existent table or temporary table (skipped from
> authorization after HIVE-20051), the authorizer will verify privileges for
> the database object.
> This would also prevent DROP TABLE IF EXISTS command failure for temporary or
> non-existing tables with Ranger. In case of temporary/non-existing table,
> empty input and output HivePrivilege Objects are sent to Ranger authroizer
> the drop table if Exists fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27195:
Description:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), the authorizer will verify privileges for the
database object.
This would also prevent DROP TABLE IF EXISTS command failure for temporary or
non-existing table with Ranger
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
was:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), the authorizer will verify privileges for the
database object.
This also prevent
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with the table object to the
> list of output objects sent for verifying privileges. This change would
> ensure that in case of a non-existent table or temporary table (skipped from
> authorization after HIVE-20051), the authorizer will verify privileges for
> the database object.
> This would also prevent DROP TABLE IF EXISTS command failure for temporary or
> non-existing table with Ranger
> the drop table if Exists fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27195:
Description:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), the authorizer will verify privileges for the
database object.
This also prevent
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
was:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), authorizer will verify privileges for database
object.
Also, In case of a temporary table drop, empty input, and output
HivePrivilegeObject are sent to the authorizer as temporary tables are skipped
from authorization.
h3. What changes were proposed in this pull request?
Authorize write actions on the database during drop table action, and add the
database object to the list of output objects sent for verifying privileges.
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with the table object to the
> list of output objects sent for verifying privileges. This change would
> ensure that in case of a non-existent table or temporary table (skipped from
> authorization after HIVE-20051), the authorizer will verify privileges for
> the database object.
> This also prevent
> the drop table if Exists fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27195:
Description:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), authorizer will verify privileges for database
object.
Also, In case of a temporary table drop, empty input, and output
HivePrivilegeObject are sent to the authorizer as temporary tables are skipped
from authorization.
h3. What changes were proposed in this pull request?
Authorize write actions on the database during drop table action, and add the
database object to the list of output objects sent for verifying privileges.
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
was:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), database object would be sent
Also, In case of a temporary table drop, empty input, and output
HivePrivilegeObject are sent to the authorizer as temporary tables are skipped
from authorization.
h3. What changes were proposed in this pull request?
Authorize write actions on the database during drop table action, and add the
database object to the list of output objects sent for verifying privileges.
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with the table object to the
> list of output objects sent for verifying privileges. This change would
> ensure that in case of a non-existent table or temporary table (skipped from
> authorization after HIVE-20051), authorizer will verify privileges for
> database object.
>
> Also, In case of a temporary table drop, empty input, and output
> HivePrivilegeObject are sent to the authorizer as temporary tables are
> skipped from authorization.
> h3. What changes were proposed in this pull request?
> Authorize write actions on the database during drop table action, and add the
> database object to the list of output objects sent for verifying privileges.
> the drop table if Exists fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27195:
Description:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of a non-existent table or temporary table (skipped from
authorization after HIVE-20051), database object would be sent
Also, In case of a temporary table drop, empty input, and output
HivePrivilegeObject are sent to the authorizer as temporary tables are skipped
from authorization.
h3. What changes were proposed in this pull request?
Authorize write actions on the database during drop table action, and add the
database object to the list of output objects sent for verifying privileges.
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
was:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of non-existent table or temporary table (skipped from
authorization )
Also, In case of a temporary table drop, empty input, and output
HivePrivilegeObject are sent to the authorizer as temporary tables are skipped
from authorization.
h3. What changes were proposed in this pull request?
Authorize write actions on the database during drop table action, and add the
database object to the list of output objects sent for verifying privileges.
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with the table object to the
> list of output objects sent for verifying privileges. This change would
> ensure that in case of a non-existent table or temporary table (skipped from
> authorization after HIVE-20051), database object would be sent
>
> Also, In case of a temporary table drop, empty input, and output
> HivePrivilegeObject are sent to the authorizer as temporary tables are
> skipped from authorization.
> h3. What changes were proposed in this pull request?
> Authorize write actions on the database during drop table action, and add the
> database object to the list of output objects sent for verifying privileges.
> the drop table if Exists fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27195:
Description:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with the table object to the
list of output objects sent for verifying privileges. This change would ensure
that in case of non-existent table or temporary table (skipped from
authorization )
Also, In case of a temporary table drop, empty input, and output
HivePrivilegeObject are sent to the authorizer as temporary tables are skipped
from authorization.
h3. What changes were proposed in this pull request?
Authorize write actions on the database during drop table action, and add the
database object to the list of output objects sent for verifying privileges.
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
was:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with table object to the list
of output objects sent for verifying privileges
Also, In case of a temporary table drop, empty input, and output
HivePrivilegeObject are sent to the authorizer as temporary tables are skipped
from authorization.
h3. What changes were proposed in this pull request?
Authorize write actions on the database during drop table action, and add the
database object to the list of output objects sent for verifying privileges.
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with the table object to the
> list of output objects sent for verifying privileges. This change would
> ensure that in case of non-existent table or temporary table (skipped from
> authorization )
>
> Also, In case of a temporary table drop, empty input, and output
> HivePrivilegeObject are sent to the authorizer as temporary tables are
> skipped from authorization.
> h3. What changes were proposed in this pull request?
> Authorize write actions on the database during drop table action, and add the
> database object to the list of output objects sent for verifying privileges.
> the drop table if Exists fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27195:
Description:
Include authorization of the database object during the "drop table" command.
Similar to "Create table", DB permissions should be verified in the case of
"drop table" too. Add the database object along with table object to the list
of output objects sent for verifying privileges
Also, In case of a temporary table drop, empty input, and output
HivePrivilegeObject are sent to the authorizer as temporary tables are skipped
from authorization.
h3. What changes were proposed in this pull request?
Authorize write actions on the database during drop table action, and add the
database object to the list of output objects sent for verifying privileges.
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
was:
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command.
> Similar to "Create table", DB permissions should be verified in the case of
> "drop table" too. Add the database object along with table object to the list
> of output objects sent for verifying privileges
>
> Also, In case of a temporary table drop, empty input, and output
> HivePrivilegeObject are sent to the authorizer as temporary tables are
> skipped from authorization.
> h3. What changes were proposed in this pull request?
> Authorize write actions on the database during drop table action, and add the
> database object to the list of output objects sent for verifying privileges.
> the drop table if Exists fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27195:
Description:
the drop table if Exists fails with HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
was:
https://issues.apache.org/jira/browse/HIVE-20051 handles skipping authorization
for temporary tables. But still, the drop table if Exists fails with
HiveAccessControlException.
Steps to Repro:
{code:java}
use test; CREATE TEMPORARY TABLE temp_table (id int);
drop table if exists test.temp_table;
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [rtrivedi] does not have [DROP] privilege on
[test/temp_table] (state=42000,code=4) {code}
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> the drop table if Exists fails with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27195) Add database authorization for drop table command
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27195:
Summary: Add database authorization for drop table command (was: Drop
table if Exists . fails during authorization for temporary
tables)
> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> https://issues.apache.org/jira/browse/HIVE-20051 handles skipping
> authorization for temporary tables. But still, the drop table if Exists fails
> with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-27398) SHOW CREATE TABLE doesn't output backticks for CLUSTERED by Col names
[
https://issues.apache.org/jira/browse/HIVE-27398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-27398 started by Riju Trivedi.
---
> SHOW CREATE TABLE doesn't output backticks for CLUSTERED by Col names
> -
>
> Key: HIVE-27398
> URL: https://issues.apache.org/jira/browse/HIVE-27398
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Minor
> Labels: pull-request-available
>
> SHOW CREATE TABLE output uses backticks for all column names and partition
> column names but does not include backticks for CLUSTERED BY column names.
> This causes ParseException during table creation when any bucket column
> identifier matches reserved keywords
> {code:java}
> CREATE TABLE `test_ts_reserved_keyword7`(
> `member_id` varchar(8),
> `plan_nr` varchar(11),
> `timestamp` timestamp,
> `shared_ind` varchar(1),
> `user_id` varchar(8))
> CLUSTERED BY (
> member_nr,
> plan_nr,
> `timestamp`)
> INTO 4 BUCKETS;
> SHOW CREATE TABLE test_ts_reserved_keyword7;
> CREATE TABLE `test_ts_reserved_keyword7`(
> `member_id` varchar(8),
> `plan_nr` varchar(11),
> `timestamp` timestamp,
> `shared_ind` varchar(1),
> `user_id` varchar(8))
> CLUSTERED BY (
> member_id,
> plan_nr,
> timestamp)
> INTO 4 BUCKETS
> ROW FORMAT
> SERDE'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'STORED AS
> INPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat';
> This fails with "Error while compiling statement: FAILED: ParseException line
> 13:0 cannot recognize input near 'timestamp' ')' 'INTO' in column name"{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27398) SHOW CREATE TABLE doesn't output backticks for CLUSTERED by Col names
[
https://issues.apache.org/jira/browse/HIVE-27398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27398:
Description:
SHOW CREATE TABLE output uses backticks for all column names and partition
column names but does not include backticks for CLUSTERED BY column names. This
causes ParseException during table creation when any bucket column identifier
matches reserved keywords
{code:java}
CREATE TABLE `test_ts_reserved_keyword7`(
`member_id` varchar(8),
`plan_nr` varchar(11),
`timestamp` timestamp,
`shared_ind` varchar(1),
`user_id` varchar(8))
CLUSTERED BY (
member_nr,
plan_nr,
`timestamp`)
INTO 4 BUCKETS;
SHOW CREATE TABLE test_ts_reserved_keyword7;
CREATE TABLE `test_ts_reserved_keyword7`(
`member_id` varchar(8),
`plan_nr` varchar(11),
`timestamp` timestamp,
`shared_ind` varchar(1),
`user_id` varchar(8))
CLUSTERED BY (
member_id,
plan_nr,
timestamp)
INTO 4 BUCKETS
ROW FORMAT
SERDE'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'STORED AS
INPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat';
This fails with "Error while compiling statement: FAILED: ParseException line
13:0 cannot recognize input near 'timestamp' ')' 'INTO' in column name"{code}
was:
SHOW CREATE TABLE output uses backticks for all column names and partition
column names but does not include backticks for CLUSTERED BY column names. This
causes ParseException during table creation when any bucket column identifier
matches reserved keywords
{code:java}
CREATE TABLE `test_ts_reserved_keyword7`(
`member_id` varchar(8),
`plan_nr` varchar(11),
`timestamp` timestamp,
`shared_ind` varchar(1),
`user_id` varchar(8))
CLUSTERED BY (
member_nr,
plan_nr,
`timestamp`)
INTO 4 BUCKETS;
SHOW CREATE TABLE test_ts_reserved_keyword7;
CREATE TABLE `test_ts_reserved_keyword7`(
`member_id` varchar(8),
`plan_nr` varchar(11),
`timestamp` timestamp,
`shared_ind` varchar(1),
`user_id` varchar(8))
CLUSTERED BY (
member_id,
plan_nr,
timestamp)
INTO 4 BUCKETS
ROW FORMAT
SERDE'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'STORED AS
INPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat';
{code}
> SHOW CREATE TABLE doesn't output backticks for CLUSTERED by Col names
> -
>
> Key: HIVE-27398
> URL: https://issues.apache.org/jira/browse/HIVE-27398
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Minor
>
> SHOW CREATE TABLE output uses backticks for all column names and partition
> column names but does not include backticks for CLUSTERED BY column names.
> This causes ParseException during table creation when any bucket column
> identifier matches reserved keywords
> {code:java}
> CREATE TABLE `test_ts_reserved_keyword7`(
> `member_id` varchar(8),
> `plan_nr` varchar(11),
> `timestamp` timestamp,
> `shared_ind` varchar(1),
> `user_id` varchar(8))
> CLUSTERED BY (
> member_nr,
> plan_nr,
> `timestamp`)
> INTO 4 BUCKETS;
> SHOW CREATE TABLE test_ts_reserved_keyword7;
> CREATE TABLE `test_ts_reserved_keyword7`(
> `member_id` varchar(8),
> `plan_nr` varchar(11),
> `timestamp` timestamp,
> `shared_ind` varchar(1),
> `user_id` varchar(8))
> CLUSTERED BY (
> member_id,
> plan_nr,
> timestamp)
> INTO 4 BUCKETS
> ROW FORMAT
> SERDE'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'STORED AS
> INPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat';
> This fails with "Error while compiling statement: FAILED: ParseException line
> 13:0 cannot recognize input near 'timestamp' ')' 'INTO' in column name"{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27398) SHOW CREATE TABLE doesn't output backticks for CLUSTERED by Col names
[
https://issues.apache.org/jira/browse/HIVE-27398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27398:
Description:
SHOW CREATE TABLE output uses backticks for all column names and partition
column names but does not include backticks for CLUSTERED BY column names. This
causes ParseException during table creation when any bucket column identifier
matches reserved keywords
{code:java}
CREATE TABLE `test_ts_reserved_keyword7`(
`member_id` varchar(8),
`plan_nr` varchar(11),
`timestamp` timestamp,
`shared_ind` varchar(1),
`user_id` varchar(8))
CLUSTERED BY (
member_nr,
plan_nr,
`timestamp`)
INTO 4 BUCKETS;
SHOW CREATE TABLE test_ts_reserved_keyword7;
CREATE TABLE `test_ts_reserved_keyword7`(
`member_id` varchar(8),
`plan_nr` varchar(11),
`timestamp` timestamp,
`shared_ind` varchar(1),
`user_id` varchar(8))
CLUSTERED BY (
member_id,
plan_nr,
timestamp)
INTO 4 BUCKETS
ROW FORMAT
SERDE'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'STORED AS
INPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat';
{code}
was:
SHOW CREATE TABLE output uses backticks for all column names and partition
column names but does not include backticks for CLUSTERED BY column names. This
causes ParseException during table creation when any bucket column identifier
matches reserved keywords when
CREATE TABLE `test_ts_reserved_keyword7`(
`member_id` varchar(8),
`plan_nr` varchar(11),
`timestamp` timestamp,
`shared_ind` varchar(1),
`user_id` varchar(8))
CLUSTERED BY (
member_nr,
plan_nr,
`timestamp`)
INTO 4 BUCKETS;
SHOW CREATE TABLE OUTPUT
CREATE TABLE `test_ts_reserved_keyword7`(
`member_id` varchar(8),
`plan_nr` varchar(11),
`timestamp` timestamp,
`shared_ind` varchar(1),
`user_id` varchar(8))
CLUSTERED BY (
member_id,
plan_nr,
timestamp)
INTO 4 BUCKETS
ROW FORMAT
SERDE'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'STORED AS
INPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat';
> SHOW CREATE TABLE doesn't output backticks for CLUSTERED by Col names
> -
>
> Key: HIVE-27398
> URL: https://issues.apache.org/jira/browse/HIVE-27398
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Minor
>
> SHOW CREATE TABLE output uses backticks for all column names and partition
> column names but does not include backticks for CLUSTERED BY column names.
> This causes ParseException during table creation when any bucket column
> identifier matches reserved keywords
> {code:java}
> CREATE TABLE `test_ts_reserved_keyword7`(
> `member_id` varchar(8),
> `plan_nr` varchar(11),
> `timestamp` timestamp,
> `shared_ind` varchar(1),
> `user_id` varchar(8))
> CLUSTERED BY (
> member_nr,
> plan_nr,
> `timestamp`)
> INTO 4 BUCKETS;
> SHOW CREATE TABLE test_ts_reserved_keyword7;
> CREATE TABLE `test_ts_reserved_keyword7`(
> `member_id` varchar(8),
> `plan_nr` varchar(11),
> `timestamp` timestamp,
> `shared_ind` varchar(1),
> `user_id` varchar(8))
> CLUSTERED BY (
> member_id,
> plan_nr,
> timestamp)
> INTO 4 BUCKETS
> ROW FORMAT
> SERDE'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'STORED AS
> INPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat';
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27398) SHOW CREATE TABLE doesn't output backticks for CLUSTERED by Col names
Riju Trivedi created HIVE-27398: --- Summary: SHOW CREATE TABLE doesn't output backticks for CLUSTERED by Col names Key: HIVE-27398 URL: https://issues.apache.org/jira/browse/HIVE-27398 Project: Hive Issue Type: Bug Components: Hive Reporter: Riju Trivedi Assignee: Riju Trivedi SHOW CREATE TABLE output uses backticks for all column names and partition column names but does not include backticks for CLUSTERED BY column names. This causes ParseException during table creation when any bucket column identifier matches reserved keywords when CREATE TABLE `test_ts_reserved_keyword7`( `member_id` varchar(8), `plan_nr` varchar(11), `timestamp` timestamp, `shared_ind` varchar(1), `user_id` varchar(8)) CLUSTERED BY ( member_nr, plan_nr, `timestamp`) INTO 4 BUCKETS; SHOW CREATE TABLE OUTPUT CREATE TABLE `test_ts_reserved_keyword7`( `member_id` varchar(8), `plan_nr` varchar(11), `timestamp` timestamp, `shared_ind` varchar(1), `user_id` varchar(8)) CLUSTERED BY ( member_id, plan_nr, timestamp) INTO 4 BUCKETS ROW FORMAT SERDE'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'STORED AS INPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27326) Hive Authorizer not receiving resource information for few alter queries causing authorization check to fail
[
https://issues.apache.org/jira/browse/HIVE-27326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi reassigned HIVE-27326:
---
Assignee: Riju Trivedi
> Hive Authorizer not receiving resource information for few alter queries
> causing authorization check to fail
>
>
> Key: HIVE-27326
> URL: https://issues.apache.org/jira/browse/HIVE-27326
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 3.1.2
>Reporter: Jai Patel
>Assignee: Riju Trivedi
>Priority: Major
>
> We have a Ranger plugin implemented for HiveService which uses the hook
> provided by the HiveService i.e. the "{*}checkPrivileges{*}" method in
> "org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizer.java"
> -
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizer.java#L163|http://example.com/].
> We do authorization based on the information provided in the *inputObjs* and
> *outputObjs* parameters.
> This *works fine* for the normal alter query like -
> {code:java}
> ALTER TABLE hr ADD COLUMNS (country VARCHAR(255)){code}
> Logs -
> {code:java}
> 2023-05-08T14:31:40,505 DEBUG [c85f84fd-85d6-4e1a-ae72-ea07323e1a93
> HiveServer2-Handler-Pool: Thread-90]
> ranger.authorization.hive.authorizer.RangerHiveAuthorizer:
> 'checkPrivileges':{'hiveOpType':ALTERTABLE_ADDCOLS,
> 'inputHObjs':['HivePrivilegeObject':{'type':TABLE_OR_VIEW, 'dbName':test,
> 'objectType':TABLE_OR_VIEW, 'objectName':hr, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER}],
> 'outputHObjs':['HivePrivilegeObject':{'type':TABLE_OR_VIEW, 'dbName':test,
> 'objectType':TABLE_OR_VIEW, 'objectName':hr, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER}],
> 'context':{'clientType':HIVESERVER2, 'commandString':ALTER TABLE hr ADD
> COLUMNS (country VARCHAR(255)), 'ipAddress':172.18.0.1,
> 'forwardedAddresses':null,
> 'sessionString':c85f84fd-85d6-4e1a-ae72-ea07323e1a93}, 'user':root,
> 'groups':[root]}
> {code}
>
> {color:#ff}*But for below alter queries, we are not getting the db and
> table information -*
> {color}Query 1 -
> {code:java}
> ALTER TABLE hr ADD CONSTRAINT unique_key_const UNIQUE (c0) DISABLE
> NOVALIDATE;{code}
> LOGS -
> {code:java}
> 2023-05-08T12:14:22,502 DEBUG [c0c66e4e-3014-4258-8e1a-7b689c2fbe6d
> HiveServer2-Handler-Pool: Thread-90]
> ranger.authorization.hive.authorizer.RangerHiveAuthorizer:
> 'checkPrivileges':{'hiveOpType':ALTERTABLE_ADDCONSTRAINT, 'inputHObjs':[],
> 'outputHObjs':[], 'context':{'clientType':HIVESERVER2, 'commandString':ALTER
> TABLE hr ADD CONSTRAINT unique_key_const1 UNIQUE (c0) DISABLE NOVALIDATE,
> 'ipAddress':172.18.0.1, 'forwardedAddresses':null, 'sessionString':c0c66{code}
> Query 2 -
> {code:java}
> ALTER TABLE temp PARTITION (c1=1) COMPACT 'minor';{code}
> Logs -
> {code:java}
> 2023-05-08T12:16:30,595 DEBUG [c0c66e4e-3014-4258-8e1a-7b689c2fbe6d
> HiveServer2-Handler-Pool: Thread-90]
> ranger.authorization.hive.authorizer.RangerHiveAuthorizer:
> 'checkPrivileges':{'hiveOpType':ALTERTABLE_COMPACT, 'inputHObjs':[],
> 'outputHObjs':[], 'context':
> {'clientType':HIVESERVER2, 'commandString':ALTER TABLE temp PARTITION (c1=1)
> COMPACT 'minor', 'ipAddress':172.18.0.1, 'forwardedAddresses':null,
> 'sessionString':c0c66e4e-3014-4258-8e1a-7b689c2fbe6d}
> , 'user':root, 'groups':[root]}
> {code}
>
>
> As you can see in the logs, we are getting empty inputHObjs and outputObjs in
> case of Alter Table Add Constraint and Partition. This is not the case for
> ALTER TABLE ADD COLUMNS and hence it works fine in that case.
> Can we fix this so as to provide proper authorization on these queries?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27195) Drop table if Exists . fails during authorization for temporary tables
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi reassigned HIVE-27195:
---
Assignee: Riju Trivedi (was: Srinivasu Majeti)
> Drop table if Exists . fails during authorization for
> temporary tables
> ---
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
>
> https://issues.apache.org/jira/browse/HIVE-20051 handles skipping
> authorization for temporary tables. But still, the drop table if Exists fails
> with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27293) Vectorization: Incorrect results with nvl for ORC table
[
https://issues.apache.org/jira/browse/HIVE-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27293:
Attachment: esource.txt
> Vectorization: Incorrect results with nvl for ORC table
> ---
>
> Key: HIVE-27293
> URL: https://issues.apache.org/jira/browse/HIVE-27293
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: esource.txt, vectorization_nvl.q
>
>
> Attached repro.q file and data file used to reproduce the issue.
> {code:java}
> Insert overwrite table etarget
> select mt.*, floor(rand() * 1) as bdata_no from (select nvl(np.client_id,'
> '),nvl(np.id_enddate,cast(0 as decimal(10,0))),nvl(np.client_gender,'
> '),nvl(np.birthday,cast(0 as decimal(10,0))),nvl(np.nationality,'
> '),nvl(np.address_zipcode,' '),nvl(np.income,cast(0 as
> decimal(15,2))),nvl(np.address,' '),nvl(np.part_date,cast(0 as int)) from
> (select * from esource where part_date = 20230414) np) mt;
> {code}
> Outcome:
> {code:java}
> select client_id,birthday,income from etarget;
> 15678 0 0.00
> 67891 19313 -1.00
> 12345 0 0.00{code}
> Expected Result :
> {code:java}
> select client_id,birthday,income from etarget;
> 12345 19613 -1.00
> 67891 19313 -1.00
> 15678 0 0.00{code}
> Disabling hive.vectorized.use.vectorized.input.format produces correct output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27293) Vectorization: Incorrect results with nvl for ORC table
[
https://issues.apache.org/jira/browse/HIVE-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27293:
Description:
Attached repro.q file and data file used to reproduce the issue.
{code:java}
Insert overwrite table etarget
select mt.*, floor(rand() * 1) as bdata_no from (select nvl(np.client_id,'
'),nvl(np.id_enddate,cast(0 as decimal(10,0))),nvl(np.client_gender,'
'),nvl(np.birthday,cast(0 as decimal(10,0))),nvl(np.nationality,'
'),nvl(np.address_zipcode,' '),nvl(np.income,cast(0 as
decimal(15,2))),nvl(np.address,' '),nvl(np.part_date,cast(0 as int)) from
(select * from esource where part_date = 20230414) np) mt;
{code}
Outcome:
{code:java}
select client_id,birthday,income from etarget;
15678 0 0.00
67891 19313 -1.00
12345 0 0.00{code}
Expected Result :
{code:java}
select client_id,birthday,income from etarget;
12345 19613 -1.00
67891 19313 -1.00
15678 0 0.00{code}
Disabling hive.vectorized.use.vectorized.input.format produces correct output.
was:
Attached repro.q file and data file used to reproduce the issue.
{code:java}
Insert overwrite table etarget
select mt.*, floor(rand() * 1) as bdata_no from (select nvl(np.client_id,'
'),nvl(np.id_enddate,cast(0 as decimal(10,0))),nvl(np.client_gender,'
'),nvl(np.birthday,cast(0 as decimal(10,0))),nvl(np.nationality,'
'),nvl(np.address_zipcode,' '),nvl(np.income,cast(0 as
decimal(15,2))),nvl(np.address,' '),nvl(np.part_date,cast(0 as int)) from
(select * from esource where part_date = 20230414) np) mt;
{code}
Outcome:
{code:java}
select client_id,birthday,income from etarget;
889004570706 0 0.00
889004570838 19880313 -1.00
889005389931 0 0.00 {code}
Expected Result :
{code:java}
select client_id,birthday,income from etarget;
889004570706 0 0.00
889004570838 19880313 -1.00
889005389931 19880613 -1.00 {code}
Disabling hive.vectorized.use.vectorized.input.format produces correct output.
> Vectorization: Incorrect results with nvl for ORC table
> ---
>
> Key: HIVE-27293
> URL: https://issues.apache.org/jira/browse/HIVE-27293
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: vectorization_nvl.q
>
>
> Attached repro.q file and data file used to reproduce the issue.
> {code:java}
> Insert overwrite table etarget
> select mt.*, floor(rand() * 1) as bdata_no from (select nvl(np.client_id,'
> '),nvl(np.id_enddate,cast(0 as decimal(10,0))),nvl(np.client_gender,'
> '),nvl(np.birthday,cast(0 as decimal(10,0))),nvl(np.nationality,'
> '),nvl(np.address_zipcode,' '),nvl(np.income,cast(0 as
> decimal(15,2))),nvl(np.address,' '),nvl(np.part_date,cast(0 as int)) from
> (select * from esource where part_date = 20230414) np) mt;
> {code}
> Outcome:
> {code:java}
> select client_id,birthday,income from etarget;
> 15678 0 0.00
> 67891 19313 -1.00
> 12345 0 0.00{code}
> Expected Result :
> {code:java}
> select client_id,birthday,income from etarget;
> 12345 19613 -1.00
> 67891 19313 -1.00
> 15678 0 0.00{code}
> Disabling hive.vectorized.use.vectorized.input.format produces correct output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27293) Vectorization: Incorrect results with nvl for ORC table
[
https://issues.apache.org/jira/browse/HIVE-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27293:
Attachment: (was: esource.txt)
> Vectorization: Incorrect results with nvl for ORC table
> ---
>
> Key: HIVE-27293
> URL: https://issues.apache.org/jira/browse/HIVE-27293
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: vectorization_nvl.q
>
>
> Attached repro.q file and data file used to reproduce the issue.
> {code:java}
> Insert overwrite table etarget
> select mt.*, floor(rand() * 1) as bdata_no from (select nvl(np.client_id,'
> '),nvl(np.id_enddate,cast(0 as decimal(10,0))),nvl(np.client_gender,'
> '),nvl(np.birthday,cast(0 as decimal(10,0))),nvl(np.nationality,'
> '),nvl(np.address_zipcode,' '),nvl(np.income,cast(0 as
> decimal(15,2))),nvl(np.address,' '),nvl(np.part_date,cast(0 as int)) from
> (select * from esource where part_date = 20230414) np) mt;
> {code}
> Outcome:
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 0 0.00 {code}
> Expected Result :
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 19880613 -1.00 {code}
> Disabling hive.vectorized.use.vectorized.input.format produces correct output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27293) Vectorization: Incorrect results with nvl for ORC table
[
https://issues.apache.org/jira/browse/HIVE-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27293:
Attachment: (was: vectorization_nvl.q)
> Vectorization: Incorrect results with nvl for ORC table
> ---
>
> Key: HIVE-27293
> URL: https://issues.apache.org/jira/browse/HIVE-27293
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: esource.txt, vectorization_nvl.q
>
>
> Attached repro.q file and data file used to reproduce the issue.
> {code:java}
> Insert overwrite table etarget
> select mt.*, floor(rand() * 1) as bdata_no from (select nvl(np.client_id,'
> '),nvl(np.id_enddate,cast(0 as decimal(10,0))),nvl(np.client_gender,'
> '),nvl(np.birthday,cast(0 as decimal(10,0))),nvl(np.nationality,'
> '),nvl(np.address_zipcode,' '),nvl(np.income,cast(0 as
> decimal(15,2))),nvl(np.address,' '),nvl(np.part_date,cast(0 as int)) from
> (select * from esource where part_date = 20230414) np) mt;
> {code}
> Outcome:
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 0 0.00 {code}
> Expected Result :
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 19880613 -1.00 {code}
> Disabling hive.vectorized.use.vectorized.input.format produces correct output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27293) Vectorization: Incorrect results with nvl for ORC table
[
https://issues.apache.org/jira/browse/HIVE-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27293:
Attachment: vectorization_nvl.q
> Vectorization: Incorrect results with nvl for ORC table
> ---
>
> Key: HIVE-27293
> URL: https://issues.apache.org/jira/browse/HIVE-27293
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: esource.txt, vectorization_nvl.q
>
>
> Attached repro.q file and data file used to reproduce the issue.
> {code:java}
> Insert overwrite table etarget
> select mt.*, floor(rand() * 1) as bdata_no from (select nvl(np.client_id,'
> '),nvl(np.id_enddate,cast(0 as decimal(10,0))),nvl(np.client_gender,'
> '),nvl(np.birthday,cast(0 as decimal(10,0))),nvl(np.nationality,'
> '),nvl(np.address_zipcode,' '),nvl(np.income,cast(0 as
> decimal(15,2))),nvl(np.address,' '),nvl(np.part_date,cast(0 as int)) from
> (select * from esource where part_date = 20230414) np) mt;
> {code}
> Outcome:
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 0 0.00 {code}
> Expected Result :
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 19880613 -1.00 {code}
> Disabling hive.vectorized.use.vectorized.input.format produces correct output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27293) Vectorization: Incorrect results with nvl for ORC table
[
https://issues.apache.org/jira/browse/HIVE-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27293:
Attachment: (was: vector_nvl.q)
> Vectorization: Incorrect results with nvl for ORC table
> ---
>
> Key: HIVE-27293
> URL: https://issues.apache.org/jira/browse/HIVE-27293
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: esource.txt, vectorization_nvl.q
>
>
> Attached repro.q file and data file used to reproduce the issue.
> {code:java}
> Insert overwrite table etarget
> select mt.*, floor(rand() * 1) as bdata_no from (select nvl(np.client_id,'
> '),nvl(np.id_enddate,cast(0 as decimal(10,0))),nvl(np.client_gender,'
> '),nvl(np.birthday,cast(0 as decimal(10,0))),nvl(np.nationality,'
> '),nvl(np.address_zipcode,' '),nvl(np.income,cast(0 as
> decimal(15,2))),nvl(np.address,' '),nvl(np.part_date,cast(0 as int)) from
> (select * from esource where part_date = 20230414) np) mt;
> {code}
> Outcome:
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 0 0.00 {code}
> Expected Result :
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 19880613 -1.00 {code}
> Disabling hive.vectorized.use.vectorized.input.format produces correct output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27293) Vectorization: Incorrect results with nvl for ORC table
[
https://issues.apache.org/jira/browse/HIVE-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27293:
Attachment: vectorization_nvl.q
> Vectorization: Incorrect results with nvl for ORC table
> ---
>
> Key: HIVE-27293
> URL: https://issues.apache.org/jira/browse/HIVE-27293
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: esource.txt, vectorization_nvl.q
>
>
> Attached repro.q file and data file used to reproduce the issue.
> {code:java}
> Insert overwrite table etarget
> select mt.*, floor(rand() * 1) as bdata_no from (select nvl(np.client_id,'
> '),nvl(np.id_enddate,cast(0 as decimal(10,0))),nvl(np.client_gender,'
> '),nvl(np.birthday,cast(0 as decimal(10,0))),nvl(np.nationality,'
> '),nvl(np.address_zipcode,' '),nvl(np.income,cast(0 as
> decimal(15,2))),nvl(np.address,' '),nvl(np.part_date,cast(0 as int)) from
> (select * from esource where part_date = 20230414) np) mt;
> {code}
> Outcome:
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 0 0.00 {code}
> Expected Result :
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 19880613 -1.00 {code}
> Disabling hive.vectorized.use.vectorized.input.format produces correct output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27293) Vectorization: Incorrect results with nvl for ORC table
[
https://issues.apache.org/jira/browse/HIVE-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27293:
Attachment: vector_nvl.q
> Vectorization: Incorrect results with nvl for ORC table
> ---
>
> Key: HIVE-27293
> URL: https://issues.apache.org/jira/browse/HIVE-27293
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: esource.txt, vector_nvl.q
>
>
> Attached repro.q file and data file used to reproduce the issue.
> {code:java}
> Insert overwrite table etarget
> select mt.*, floor(rand() * 1) as bdata_no from (select nvl(np.client_id,'
> '),nvl(np.id_enddate,cast(0 as decimal(10,0))),nvl(np.client_gender,'
> '),nvl(np.birthday,cast(0 as decimal(10,0))),nvl(np.nationality,'
> '),nvl(np.address_zipcode,' '),nvl(np.income,cast(0 as
> decimal(15,2))),nvl(np.address,' '),nvl(np.part_date,cast(0 as int)) from
> (select * from esource where part_date = 20230414) np) mt;
> {code}
> Outcome:
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 0 0.00 {code}
> Expected Result :
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 19880613 -1.00 {code}
> Disabling hive.vectorized.use.vectorized.input.format produces correct output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27293) Vectorization: Incorrect results with nvl for ORC table
[
https://issues.apache.org/jira/browse/HIVE-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27293:
Attachment: esource.txt
> Vectorization: Incorrect results with nvl for ORC table
> ---
>
> Key: HIVE-27293
> URL: https://issues.apache.org/jira/browse/HIVE-27293
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Riju Trivedi
>Priority: Major
> Attachments: esource.txt
>
>
> Attached repro.q file and data file used to reproduce the issue.
> {code:java}
> Insert overwrite table etarget
> select mt.*, floor(rand() * 1) as bdata_no from (select nvl(np.client_id,'
> '),nvl(np.id_enddate,cast(0 as decimal(10,0))),nvl(np.client_gender,'
> '),nvl(np.birthday,cast(0 as decimal(10,0))),nvl(np.nationality,'
> '),nvl(np.address_zipcode,' '),nvl(np.income,cast(0 as
> decimal(15,2))),nvl(np.address,' '),nvl(np.part_date,cast(0 as int)) from
> (select * from esource where part_date = 20230414) np) mt;
> {code}
> Outcome:
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 0 0.00 {code}
> Expected Result :
> {code:java}
> select client_id,birthday,income from etarget;
> 889004570706 0 0.00
> 889004570838 19880313 -1.00
> 889005389931 19880613 -1.00 {code}
> Disabling hive.vectorized.use.vectorized.input.format produces correct output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27293) Vectorization: Incorrect results with nvl for ORC table
Riju Trivedi created HIVE-27293:
---
Summary: Vectorization: Incorrect results with nvl for ORC table
Key: HIVE-27293
URL: https://issues.apache.org/jira/browse/HIVE-27293
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 4.0.0-alpha-2
Reporter: Riju Trivedi
Attachments: esource.txt
Attached repro.q file and data file used to reproduce the issue.
{code:java}
Insert overwrite table etarget
select mt.*, floor(rand() * 1) as bdata_no from (select nvl(np.client_id,'
'),nvl(np.id_enddate,cast(0 as decimal(10,0))),nvl(np.client_gender,'
'),nvl(np.birthday,cast(0 as decimal(10,0))),nvl(np.nationality,'
'),nvl(np.address_zipcode,' '),nvl(np.income,cast(0 as
decimal(15,2))),nvl(np.address,' '),nvl(np.part_date,cast(0 as int)) from
(select * from esource where part_date = 20230414) np) mt;
{code}
Outcome:
{code:java}
select client_id,birthday,income from etarget;
889004570706 0 0.00
889004570838 19880313 -1.00
889005389931 0 0.00 {code}
Expected Result :
{code:java}
select client_id,birthday,income from etarget;
889004570706 0 0.00
889004570838 19880313 -1.00
889005389931 19880613 -1.00 {code}
Disabling hive.vectorized.use.vectorized.input.format produces correct output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-24815) Remove "IDXS" Table from Metastore Schema
[ https://issues.apache.org/jira/browse/HIVE-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi reassigned HIVE-24815: --- Assignee: Riju Trivedi (was: Hunter Logan) > Remove "IDXS" Table from Metastore Schema > - > > Key: HIVE-24815 > URL: https://issues.apache.org/jira/browse/HIVE-24815 > Project: Hive > Issue Type: Improvement > Components: Metastore, Standalone Metastore >Affects Versions: 3.1.0, 3.0.0, 3.1.1, 3.1.2, 3.2.0, 4.0.0 >Reporter: Hunter Logan >Assignee: Riju Trivedi >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > In Hive 3 the rarely used "INDEXES" was removed from the DDL > https://issues.apache.org/jira/browse/HIVE-18448 > > There are a few issues here: > # The Standalone-Metastore schema for Hive 3+ all include the "IDXS" table, > which has no function. > ** > [https://github.com/apache/hive/tree/master/standalone-metastore/metastore-server/src/main/sql/mysql] > # The upgrade schemas from 2.x -> 3.x do not do any cleanup of the IDXS table > ** If a user used the "INDEXES" feature in 2.x and then upgrades their > metastore to 3.x+ they cannot drop any table that has an index on it due to > "IDXS_FK1" constraint since the TBLS entry is referenced in the IDXS table > ** Since INDEX is no longer in the DDL they cannot run any command from Hive > to drop the index. > ** Users can manually connect to the metastore and either drop the IDXS > table or the foreign key constraint > > Since indexes provide no benefits in Hive 3+ it should be fine to drop them > completely in the schema upgrade scripts. At the very least the 2.x -> 3.x+ > scripts should drop the fk constraint. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27195) Drop table if Exists . fails during authorization for temporary tables
[
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi reassigned HIVE-27195:
---
> Drop table if Exists . fails during authorization for
> temporary tables
> ---
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
>
> https://issues.apache.org/jira/browse/HIVE-20051 handles skipping
> authorization for temporary tables. But still, the drop table if Exists fails
> with HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [rtrivedi] does not have [DROP] privilege on
> [test/temp_table] (state=42000,code=4) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27181) Remove RegexSerDe from hive-contrib, Upgrade should update changed FQN for RegexSerDe in HMS DB
[
https://issues.apache.org/jira/browse/HIVE-27181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27181:
Description: Remove o{*}rg.apache.hadoop.hive.contrib.serde2.RegexSerDe{*}
from hive-contrib. Fix tests to use the new Serde class
org.apache.hadoop.hive.serde2.RegexSerDe. Hive Upgrade schema script can update
the SERDES table to alter the class name to the new class name, the old tables
would work automatically.
> Remove RegexSerDe from hive-contrib, Upgrade should update changed FQN for
> RegexSerDe in HMS DB
> ---
>
> Key: HIVE-27181
> URL: https://issues.apache.org/jira/browse/HIVE-27181
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
>
> Remove o{*}rg.apache.hadoop.hive.contrib.serde2.RegexSerDe{*} from
> hive-contrib. Fix tests to use the new Serde class
> org.apache.hadoop.hive.serde2.RegexSerDe. Hive Upgrade schema script can
> update the SERDES table to alter the class name to the new class name, the
> old tables would work automatically.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27180) Remove JsonSerde from hcatalog, Upgrade should update changed FQN for JsonSerDe in HMS DB
[
https://issues.apache.org/jira/browse/HIVE-27180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27180:
Description: As Hcatalog JsonSerDe uses the "serde2" version as a back end,
Remove o{*}rg.apache.hive.hcatalog.data.JsonSerDe{*} from hive-hcatalog. Fix
tests to use the new Serde class org.apache.hadoop.hive.serde2.JsonSerDe. Hive
Upgrade schema script can update the SERDES table to alter the class name to
the new class name, the old tables would work automatically.
> Remove JsonSerde from hcatalog, Upgrade should update changed FQN for
> JsonSerDe in HMS DB
> --
>
> Key: HIVE-27180
> URL: https://issues.apache.org/jira/browse/HIVE-27180
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
>
> As Hcatalog JsonSerDe uses the "serde2" version as a back end, Remove
> o{*}rg.apache.hive.hcatalog.data.JsonSerDe{*} from hive-hcatalog. Fix tests
> to use the new Serde class org.apache.hadoop.hive.serde2.JsonSerDe. Hive
> Upgrade schema script can update the SERDES table to alter the class name to
> the new class name, the old tables would work automatically.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27181) Remove RegexSerDe from hive-contrib, Upgrade should update changed FQN for RegexSerDe in HMS DB
[ https://issues.apache.org/jira/browse/HIVE-27181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi reassigned HIVE-27181: --- Assignee: Riju Trivedi > Remove RegexSerDe from hive-contrib, Upgrade should update changed FQN for > RegexSerDe in HMS DB > --- > > Key: HIVE-27181 > URL: https://issues.apache.org/jira/browse/HIVE-27181 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27180) Remove JsonSerde from hcatalog, Upgrade should update changed FQN for JsonSerDe in HMS DB
[ https://issues.apache.org/jira/browse/HIVE-27180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi reassigned HIVE-27180: --- > Remove JsonSerde from hcatalog, Upgrade should update changed FQN for > JsonSerDe in HMS DB > -- > > Key: HIVE-27180 > URL: https://issues.apache.org/jira/browse/HIVE-27180 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26276) Fix package to org.apache.hadoop.hive.serde2 for JsonSerDe & RegexSerDe in HMS DB
[
https://issues.apache.org/jira/browse/HIVE-26276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi reassigned HIVE-26276:
---
Assignee: Riju Trivedi
> Fix package to org.apache.hadoop.hive.serde2 for JsonSerDe & RegexSerDe in
> HMS DB
> -
>
> Key: HIVE-26276
> URL: https://issues.apache.org/jira/browse/HIVE-26276
> Project: Hive
> Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Riju Trivedi
>Priority: Major
>
> Similar to HIVE-24770, JsonSerDe & RegexSerDe should be updated to newer
> package
> {code:java}
> // Avoid dependency of hive-hcatalog.jar
> Old - org.apache.hive.hcatalog.data.JsonSerDe
> New - org.apache.hadoop.hive.serde2.JsonSerDe
> // Avoid dependency of hive-contrib.jar
> Old - org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> New - org.apache.hadoop.hive.serde2.RegexSerDe
> {code}
> This should be handled in upgrade flow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27113) Increasing default for hive.thrift.client.max.message.size to 2 GB
[
https://issues.apache.org/jira/browse/HIVE-27113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698231#comment-17698231
]
Riju Trivedi commented on HIVE-27113:
-
Thrift message size can go beyond 1g for wide tables with too many partitions
(5k+), Hence increasing the default to max value 2 GB.
> Increasing default for hive.thrift.client.max.message.size to 2 GB
> --
>
> Key: HIVE-27113
> URL: https://issues.apache.org/jira/browse/HIVE-27113
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
>
> HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size",
> "1gb",
> new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
> "Thrift client configuration for max message size. 0 or -1 will use
> the default defined in the Thrift " +
> "library. The upper limit is 2147483648 bytes (or 2gb).")
> Documentation on the help suggests setting 2147483648 while Integer Max is
> 2147483647. So, it actually becomes -1 and gets set to thrift default limit
> (100 MB)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27113) Increasing default for hive.thrift.client.max.message.size to 2 GB
[
https://issues.apache.org/jira/browse/HIVE-27113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27113:
Summary: Increasing default for hive.thrift.client.max.message.size to 2 GB
(was: Documentation for hive.thrift.client.max.message.size config needs to be
corrected)
> Increasing default for hive.thrift.client.max.message.size to 2 GB
> --
>
> Key: HIVE-27113
> URL: https://issues.apache.org/jira/browse/HIVE-27113
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
>
> HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size",
> "1gb",
> new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
> "Thrift client configuration for max message size. 0 or -1 will use
> the default defined in the Thrift " +
> "library. The upper limit is 2147483648 bytes (or 2gb).")
> Documentation on the help suggests setting 2147483648 while Integer Max is
> 2147483647. So, it actually becomes -1 and gets set to thrift default limit
> (100 MB)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27113) Documentation for hive.thrift.client.max.message.size config needs to be corrected
[
https://issues.apache.org/jira/browse/HIVE-27113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-27113:
Component/s: Hive
> Documentation for hive.thrift.client.max.message.size config needs to be
> corrected
> --
>
> Key: HIVE-27113
> URL: https://issues.apache.org/jira/browse/HIVE-27113
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
>
> HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size",
> "1gb",
> new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
> "Thrift client configuration for max message size. 0 or -1 will use
> the default defined in the Thrift " +
> "library. The upper limit is 2147483648 bytes (or 2gb).")
> Documentation on the help suggests setting 2147483648 while Integer Max is
> 2147483647. So, it actually becomes -1 and gets set to thrift default limit
> (100 MB)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27113) Documentation for hive.thrift.client.max.message.size config needs to be corrected
[
https://issues.apache.org/jira/browse/HIVE-27113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi reassigned HIVE-27113:
---
> Documentation for hive.thrift.client.max.message.size config needs to be
> corrected
> --
>
> Key: HIVE-27113
> URL: https://issues.apache.org/jira/browse/HIVE-27113
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
>
> HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size",
> "1gb",
> new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
> "Thrift client configuration for max message size. 0 or -1 will use
> the default defined in the Thrift " +
> "library. The upper limit is 2147483648 bytes (or 2gb).")
> Documentation on the help suggests setting 2147483648 while Integer Max is
> 2147483647. So, it actually becomes -1 and gets set to thrift default limit
> (100 MB)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27011) Default value of PartitionManagementTask frequency should be set to higher value
[ https://issues.apache.org/jira/browse/HIVE-27011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi resolved HIVE-27011. - Fix Version/s: 4.0.0 Resolution: Fixed > Default value of PartitionManagementTask frequency should be set to higher > value > > > Key: HIVE-27011 > URL: https://issues.apache.org/jira/browse/HIVE-27011 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Fix For: 4.0.0 > > > Default for "metastore.partition.management.task.frequency" is 5 mins, less > than ideal for Prod scenarios. When there are a large number of > databases/tables, it takes a lot of time for PartitionManagementTask to scan > all tables and partitions and doesn't complete within 5 mins. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27011) Default value of PartitionManagementTask frequency should be set to higher value
[ https://issues.apache.org/jira/browse/HIVE-27011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi reassigned HIVE-27011: --- > Default value of PartitionManagementTask frequency should be set to higher > value > > > Key: HIVE-27011 > URL: https://issues.apache.org/jira/browse/HIVE-27011 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > > Default for "metastore.partition.management.task.frequency" is 5 mins, less > than ideal for Prod scenarios. When there are a large number of > databases/tables, it takes a lot of time for PartitionManagementTask to scan > all tables and partitions and doesn't complete within 5 mins. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26898) Split Notification logging so that busy clusters can have better performance
[ https://issues.apache.org/jira/browse/HIVE-26898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682238#comment-17682238 ] Riju Trivedi commented on HIVE-26898: - [~ayushtkn] Thanks for pointing this out. This comment was meant for another internal jira, mistakenly posted here. > Split Notification logging so that busy clusters can have better performance > > > Key: HIVE-26898 > URL: https://issues.apache.org/jira/browse/HIVE-26898 > Project: Hive > Issue Type: New Feature >Reporter: Taraka Rama Rao Lethavadla >Priority: Major > > For DDL & DML events are logged into notifications log table and those get > cleaned as soon as ttl got expired. > In most of the busy clusters, the notification log is growing even though > cleaner is running and kept on cleaning the events. It means the rate of Hive > db operations are very high compared to rate at which cleaning is happening. > So any query on this table is becoming bottle neck at backend DB causing slow > response > The proposal is to split the notification log table in to multiple tables > like > notification_log_dml - for all DML queries > notification_log_insert - for all insert queries > .. > etc. > > So that load on that single table gets reduced improving the performance of > the backend db as well as Hive -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26898) Split Notification logging so that busy clusters can have better performance
[ https://issues.apache.org/jira/browse/HIVE-26898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682210#comment-17682210 ] Riju Trivedi commented on HIVE-26898: - [~tarak271] Do we see this notification_log table growing and OOM issues In recent versions 717, SP1 and SP2? Improvements like loading notifications in batches (HIVE-19430) and handling batches in different transactions (HIVE-24432) are already included in these versions. > Split Notification logging so that busy clusters can have better performance > > > Key: HIVE-26898 > URL: https://issues.apache.org/jira/browse/HIVE-26898 > Project: Hive > Issue Type: New Feature >Reporter: Taraka Rama Rao Lethavadla >Priority: Major > > For DDL & DML events are logged into notifications log table and those get > cleaned as soon as ttl got expired. > In most of the busy clusters, the notification log is growing even though > cleaner is running and kept on cleaning the events. It means the rate of Hive > db operations are very high compared to rate at which cleaning is happening. > So any query on this table is becoming bottle neck at backend DB causing slow > response > The proposal is to split the notification log table in to multiple tables > like > notification_log_dml - for all DML queries > notification_log_insert - for all insert queries > .. > etc. > > So that load on that single table gets reduced improving the performance of > the backend db as well as Hive -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-24070) ObjectStore.cleanWriteNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-24070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195184#comment-17195184 ] Riju Trivedi edited comment on HIVE-24070 at 9/14/20, 8:54 AM: --- [~rameshkumar] [~nareshpr] I think we are trying to address same issues in both of these jiras HIVE-22290 was (Author: rtrivedi12): [~rameshkumar] [~nareshpr] I think we are trying address same issues in both of these jiras [HIVE-22290|https://issues.apache.org/jira/browse/HIVE-22290] > ObjectStore.cleanWriteNotificationEvents OutOfMemory on large number of > pending events > -- > > Key: HIVE-24070 > URL: https://issues.apache.org/jira/browse/HIVE-24070 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Fix For: 4.0.0 > > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanWriteNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. > Similar to https://issues.apache.org/jira/browse/HIVE-19430 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24070) ObjectStore.cleanWriteNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-24070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195184#comment-17195184 ] Riju Trivedi commented on HIVE-24070: - [~rameshkumar] [~nareshpr] I think we are trying address same issues in both of these jiras [HIVE-22290|https://issues.apache.org/jira/browse/HIVE-22290] > ObjectStore.cleanWriteNotificationEvents OutOfMemory on large number of > pending events > -- > > Key: HIVE-24070 > URL: https://issues.apache.org/jira/browse/HIVE-24070 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Fix For: 4.0.0 > > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanWriteNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. > Similar to https://issues.apache.org/jira/browse/HIVE-19430 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23471) Statement.executeUpdate() does not return correct affected records causing "No such lock"
[
https://issues.apache.org/jira/browse/HIVE-23471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi reassigned HIVE-23471:
---
> Statement.executeUpdate() does not return correct affected records causing
> "No such lock"
> -
>
> Key: HIVE-23471
> URL: https://issues.apache.org/jira/browse/HIVE-23471
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Denys Kuzmenko
>Priority: Critical
>
> In TxnHandler.acquire() call , Statement.executeUpdate() does not return
> correct number of records updated in HIVE_LOCKS table as requested number of
> locks.
> This results in error "*Couldn't find a lock we just created! No such
> lock(s)*" as acquire is rolled back.
> {code:java}
> int rc = stmt.executeUpdate(s);
> if (rc < locksBeingChecked.size()) {
> LOG.debug("Going to rollback acquire(Connection dbConn, Statement stmt,
> List locksBeingChecked)");
> dbConn.rollback();
> /*select all locks for this ext ID and see which ones are missing*/
> StringBuilder sb = new StringBuilder("No such lock(s): (" +
> JavaUtils.lockIdToString(extLockId) + ":");
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23339) SBA does not check permissions for DB location specified in Create database query
[
https://issues.apache.org/jira/browse/HIVE-23339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096477#comment-17096477
]
Riju Trivedi commented on HIVE-23339:
-
During doAuthorization() call for CreateDatabase operation , authorize is
invoked only with read and write Privileges for the operation.
{code:java}
public void authorize(Privilege[] readRequiredPriv, Privilege[]
writeRequiredPriv)
throws HiveException, AuthorizationException {
{code}
In StorageBasedAuth, we only call checkPermissions() for root warehouse dir
(hive.metastore.warehouse.dir) and not the specified location. So, any user who
does not have access to directory will be able to create database if they have
access to warehouse path.
{code:java}
Path root = null;
try {
initWh();
root = wh.getWhRoot();
authorize(root, readRequiredPriv, writeRequiredPriv);{code}
> SBA does not check permissions for DB location specified in Create database
> query
> -
>
> Key: HIVE-23339
> URL: https://issues.apache.org/jira/browse/HIVE-23339
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Shubham Chaurasia
>Priority: Critical
>
> With doAs=true and StorageBasedAuthorization provider, create database with
> specific location succeeds even if user doesn't have access to that path.
>
> {code:java}
> hadoop fs -ls -d /tmp/cannot_write
> drwx-- - hive hadoop 0 2020-04-01 22:53 /tmp/cannot_write
> create a database under /tmp/cannot_write. We would expect it to fail, but is
> actually created successfully with "hive" as the owner:
> rtrivedi@bdp01:~> beeline -e "create database rtrivedi_1 location
> '/tmp/cannot_write/rtrivedi_1'"
> INFO : OK
> No rows affected (0.116 seconds)
> hive@hpchdd2e:~> hadoop fs -ls /tmp/cannot_write
> Found 1 items
> drwx-- - hive hadoop 0 2020-04-01 23:05 /tmp/cannot_write/rtrivedi_1
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23339) SBA does not check permissions for DB location specified in Create database query
[
https://issues.apache.org/jira/browse/HIVE-23339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi reassigned HIVE-23339:
---
> SBA does not check permissions for DB location specified in Create database
> query
> -
>
> Key: HIVE-23339
> URL: https://issues.apache.org/jira/browse/HIVE-23339
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Shubham Chaurasia
>Priority: Critical
>
> With doAs=true and StorageBasedAuthorization provider, create database with
> specific location succeeds even if user doesn't have access to that path.
>
> {code:java}
> hadoop fs -ls -d /tmp/cannot_write
> drwx-- - hive hadoop 0 2020-04-01 22:53 /tmp/cannot_write
> create a database under /tmp/cannot_write. We would expect it to fail, but is
> actually created successfully with "hive" as the owner:
> rtrivedi@bdp01:~> beeline -e "create database rtrivedi_1 location
> '/tmp/cannot_write/rtrivedi_1'"
> INFO : OK
> No rows affected (0.116 seconds)
> hive@hpchdd2e:~> hadoop fs -ls /tmp/cannot_write
> Found 1 items
> drwx-- - hive hadoop 0 2020-04-01 23:05 /tmp/cannot_write/rtrivedi_1
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Attachment: HIVE-23058.3.patch > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE-23058.2.patch, HIVE-23058.3.patch, > HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Attachment: (was: HIVE-23058.3.patch) > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE-23058.2.patch, HIVE-23058.3.patch, > HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Attachment: HIVE-23058.3.patch > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE-23058.2.patch, HIVE-23058.3.patch, > HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Attachment: (was: HIVE-23058.3.patch) > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE-23058.2.patch, HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Attachment: HIVE-23058.3.patch > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE-23058.2.patch, HIVE-23058.3.patch, > HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078891#comment-17078891 ] Riju Trivedi commented on HIVE-23058: - [~lpinter] All the above 3 tests failure seems unrelated to the patch . Can you please confirm. > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE-23058.2.patch, HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Attachment: HIVE-23058.2.patch > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE-23058.2.patch, HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Status: Patch Available (was: Open) > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Attachment: HIVE_23058.1.patch > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Attachment: HIVE_23058.patch > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi reassigned HIVE-23058: --- > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22452) CTAS query failure at DDL task stage doesn't clean out the target directory
[ https://issues.apache.org/jira/browse/HIVE-22452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi reassigned HIVE-22452: --- Assignee: Marta Kuczora (was: Riju Trivedi) > CTAS query failure at DDL task stage doesn't clean out the target directory > --- > > Key: HIVE-22452 > URL: https://issues.apache.org/jira/browse/HIVE-22452 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0, 3.1.2 >Reporter: Riju Trivedi >Assignee: Marta Kuczora >Priority: Major > > CTAS query failure at DDL task stage due to HMS connection issue leaves the > output file in target directory. Since DDL task stage happens after Tez DAG > completion and MOVE Task , output file gets already moved to target > directory and does not get cleaned up after the query failure. > Re-executing the same query causes a duplicate file under table location > hence duplicate data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22452) CTAS query failure at DDL task stage doesn't clean out the target directory
[ https://issues.apache.org/jira/browse/HIVE-22452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi reassigned HIVE-22452: --- > CTAS query failure at DDL task stage doesn't clean out the target directory > --- > > Key: HIVE-22452 > URL: https://issues.apache.org/jira/browse/HIVE-22452 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > > CTAS query failure at DDL task stage due to HMS connection issue leaves the > output file in target directory. Since DDL task stage happens after Tez DAG > completion and MOVE Task , output file gets already moved to target > directory and does not get cleaned up after the query failure. > Re-executing the same query causes a duplicate file under table location > hence duplicate data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22208) Column name with reserved keyword is unescaped when query including join on table with mask column is re-written
[
https://issues.apache.org/jira/browse/HIVE-22208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi reassigned HIVE-22208:
---
Assignee: Jesus Camacho Rodriguez
> Column name with reserved keyword is unescaped when query including join on
> table with mask column is re-written
>
>
> Key: HIVE-22208
> URL: https://issues.apache.org/jira/browse/HIVE-22208
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Riju Trivedi
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
>
> Join query involving table with mask column and other having reserved
> keyword as column name fails with SemanticException during parsing re-written
> query :
> Original Query :
> {code:java}
> select a.`date`, b.nm
> from sample_keyword a
> join sample_mask b
> on b.id = a.id;
> {code}
> Re-written Query :
>
> {code:java}
> select a.date, b.nm
> from sample_keyword a
> join (SELECT `id`, CAST(mask_hash(nm) AS string) AS `nm`,
> BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID FROM
> `default`.`sample_mask` )`b`
> on b.id = a.id;
> {code}
> Re-written query does not have escape quotes for date column which cause
> SemanticException while parsing :
> {code:java}
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near 'a' '.' 'date' in selection target
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(
> SemanticAnalyzer.java:12084)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
> SemanticAnalyzer.java:12298)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
> CalcitePlanner.java:360)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
> BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22208) Column name with reserved keyword is unescaped when query including join on table with mask column is re-written
[
https://issues.apache.org/jira/browse/HIVE-22208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-22208:
Description:
Join query involving table with mask column and other having reserved keyword
as column name fails with SemanticException during parsing re-written query :
Original Query :
{code:java}
select a.`date`, b.nm
from sample_keyword a
join sample_mask b
on b.id = a.id;
{code}
Re-written Query :
{code:java}
select a.date, b.nm
from sample_keyword a
join (SELECT `id`, CAST(mask_hash(nm) AS string) AS `nm`,
BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID FROM
`default`.`sample_mask` )`b`
on b.id = a.id;
{code}
Re-written query does not have escape quotes for date column which cause
SemanticException while parsing :
{code:java}
org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize input
near 'a' '.' 'date' in selection target
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(
SemanticAnalyzer.java:12084)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
SemanticAnalyzer.java:12298)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
CalcitePlanner.java:360)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
{code}
was:
Join query involving table with mask column and other having reserved keyword
as column name fails with SemanticException during parsing re-written query :
Original Query :
{code:java}
select a.`date`, b.nm
from sample_keyword a
join sample_mask b
on b.id = a.id;
{code}
Re-written Query :
{code:java}
select a.date, b.nm
from sample_keyword a
join (SELECT `id`, CAST(mask_hash(nm) AS string) AS `nm`,
BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID FROM
`default`.`sample_mask` )`b`
on b.id = a.id;
{code}
Re-written query does not have escape quotes for date column which cause
SemanticException while parsing :
{code:java}
org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize input
near 'a' '.' 'date' in selection target
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(
SemanticAnalyzer.java:12084)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
SemanticAnalyzer.java:12298)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
CalcitePlanner.java:360)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
{code}
> Column name with reserved keyword is unescaped when query including join on
> table with mask column is re-written
>
>
> Key: HIVE-22208
> URL: https://issues.apache.org/jira/browse/HIVE-22208
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Riju Trivedi
>Priority: Critical
>
> Join query involving table with mask column and other having reserved
> keyword as column name fails with SemanticException during parsing re-written
> query :
> Original Query :
> {code:java}
> select a.`date`, b.nm
> from sample_keyword a
> join sample_mask b
> on b.id = a.id;
> {code}
> Re-written Query :
>
> {code:java}
> select a.date, b.nm
> from sample_keyword a
> join (SELECT `id`, CAST(mask_hash(nm) AS string) AS `nm`,
> BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID FROM
> `default`.`sample_mask` )`b`
> on b.id = a.id;
> {code}
> Re-written query does not have escape quotes for date column which cause
> SemanticException while parsing :
> {code:java}
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near 'a' '.' 'date' in selection target
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(
> SemanticAnalyzer.java:12084)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
> SemanticAnalyzer.java:12298)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
> CalcitePlanner.java:360)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
>
[jira] [Updated] (HIVE-22208) Column name with reserved keyword is unescaped when query including join on table with mask column is re-written
[
https://issues.apache.org/jira/browse/HIVE-22208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-22208:
Summary: Column name with reserved keyword is unescaped when query
including join on table with mask column is re-written (was: Column name with
reserved keyword is unescaped when query includes join on table with mask
column is re-written)
> Column name with reserved keyword is unescaped when query including join on
> table with mask column is re-written
>
>
> Key: HIVE-22208
> URL: https://issues.apache.org/jira/browse/HIVE-22208
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Riju Trivedi
>Priority: Critical
>
> Join query involving table with mask column and other having reserved
> keyword as column name fails with SemanticException during parsing re-written
> query :
> Original Query :
> {code:java}
> select a.`date`, b.nm
> from sample_keyword a
> join sample_mask b
> on b.id = a.id;
> {code}
> Re-written Query :
>
> {code:java}
> select a.date, b.nm
> from sample_keyword a
> join (SELECT `id`, CAST(mask_hash(nm) AS string) AS `nm`,
> BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID FROM
> `default`.`sample_mask` )`b`
> on b.id = a.id;
> {code}
> Re-written query does not have escape quotes for date column which cause
> SemanticException while parsing :
>
>
> {code:java}
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near 'a' '.' 'date' in selection target
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(
> SemanticAnalyzer.java:12084)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
> SemanticAnalyzer.java:12298)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
> CalcitePlanner.java:360)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
> BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-21061) CTAS query fails with IllegalStateException for empty source
[
https://issues.apache.org/jira/browse/HIVE-21061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829952#comment-16829952
]
Riju Trivedi commented on HIVE-21061:
-
Failed test seems to be unrelated.
> CTAS query fails with IllegalStateException for empty source
>
>
> Key: HIVE-21061
> URL: https://issues.apache.org/jira/browse/HIVE-21061
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE-21061.1.patch, HIVE-21061.2.patch,
> HIVE-21061.2.patch, HIVE-21061.patch
>
>
> Creating a table using CTAS from an empty source table with predicate
> condition evaluating to False
> {code}
> create table testctas1 (id int);
> create table testctas2 as select * from testctas1 where 1=2;
> {code}
> Fails with below exception:
> {code}
> Caused by: java.lang.IllegalStateException: null
> at com.google.common.base.Preconditions.checkState(Preconditions.java:159)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.verifyAndSetVectorPartDesc(Vectorizer.java:1312)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateInputFormatAndSchemaEvolution(Vectorizer.java:1654)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:1865)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:1109)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:961)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:2442)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:717)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:258)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12443)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1863)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1810)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1805)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
> ... 36 more
> {code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21061) CTAS query fails with IllegalStateException for empty source
[
https://issues.apache.org/jira/browse/HIVE-21061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-21061:
Attachment: HIVE-21061.2.patch
> CTAS query fails with IllegalStateException for empty source
>
>
> Key: HIVE-21061
> URL: https://issues.apache.org/jira/browse/HIVE-21061
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE-21061.1.patch, HIVE-21061.2.patch,
> HIVE-21061.2.patch, HIVE-21061.patch
>
>
> Creating a table using CTAS from an empty source table with predicate
> condition evaluating to False
> {code}
> create table testctas1 (id int);
> create table testctas2 as select * from testctas1 where 1=2;
> {code}
> Fails with below exception:
> {code}
> Caused by: java.lang.IllegalStateException: null
> at com.google.common.base.Preconditions.checkState(Preconditions.java:159)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.verifyAndSetVectorPartDesc(Vectorizer.java:1312)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateInputFormatAndSchemaEvolution(Vectorizer.java:1654)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:1865)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:1109)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:961)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:2442)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:717)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:258)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12443)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1863)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1810)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1805)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
> ... 36 more
> {code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21061) CTAS query fails with IllegalStateException for empty source
[
https://issues.apache.org/jira/browse/HIVE-21061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-21061:
Attachment: HIVE-21061.2.patch
> CTAS query fails with IllegalStateException for empty source
>
>
> Key: HIVE-21061
> URL: https://issues.apache.org/jira/browse/HIVE-21061
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE-21061.1.patch, HIVE-21061.2.patch, HIVE-21061.patch
>
>
> Creating a table using CTAS from an empty source table with predicate
> condition evaluating to False
> {code}
> create table testctas1 (id int);
> create table testctas2 as select * from testctas1 where 1=2;
> {code}
> Fails with below exception:
> {code}
> Caused by: java.lang.IllegalStateException: null
> at com.google.common.base.Preconditions.checkState(Preconditions.java:159)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.verifyAndSetVectorPartDesc(Vectorizer.java:1312)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateInputFormatAndSchemaEvolution(Vectorizer.java:1654)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:1865)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:1109)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:961)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:2442)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:717)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:258)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12443)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1863)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1810)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1805)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
> ... 36 more
> {code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21061) CTAS query fails with IllegalStateException for empty source
[
https://issues.apache.org/jira/browse/HIVE-21061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riju Trivedi updated HIVE-21061:
Status: Patch Available (was: Open)
> CTAS query fails with IllegalStateException for empty source
>
>
> Key: HIVE-21061
> URL: https://issues.apache.org/jira/browse/HIVE-21061
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE-21061.1.patch, HIVE-21061.2.patch, HIVE-21061.patch
>
>
> Creating a table using CTAS from an empty source table with predicate
> condition evaluating to False
> {code}
> create table testctas1 (id int);
> create table testctas2 as select * from testctas1 where 1=2;
> {code}
> Fails with below exception:
> {code}
> Caused by: java.lang.IllegalStateException: null
> at com.google.common.base.Preconditions.checkState(Preconditions.java:159)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.verifyAndSetVectorPartDesc(Vectorizer.java:1312)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateInputFormatAndSchemaEvolution(Vectorizer.java:1654)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:1865)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:1109)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:961)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:2442)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:717)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:258)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12443)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1863)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1810)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1805)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
> ... 36 more
> {code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21061) CTAS query fails with IllegalStateException for empty source
[
https://issues.apache.org/jira/browse/HIVE-21061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829148#comment-16829148
]
Riju Trivedi commented on HIVE-21061:
-
[~vgumashta] Added test case.
> CTAS query fails with IllegalStateException for empty source
>
>
> Key: HIVE-21061
> URL: https://issues.apache.org/jira/browse/HIVE-21061
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE-21061.1.patch, HIVE-21061.patch
>
>
> Creating a table using CTAS from an empty source table with predicate
> condition evaluating to False
> {code}
> create table testctas1 (id int);
> create table testctas2 as select * from testctas1 where 1=2;
> {code}
> Fails with below exception:
> {code}
> Caused by: java.lang.IllegalStateException: null
> at com.google.common.base.Preconditions.checkState(Preconditions.java:159)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.verifyAndSetVectorPartDesc(Vectorizer.java:1312)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateInputFormatAndSchemaEvolution(Vectorizer.java:1654)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:1865)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:1109)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:961)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:2442)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:717)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:258)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12443)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1863)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1810)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1805)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
> ... 36 more
> {code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
