[jira] [Work logged] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?focusedWorklogId=813832=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813832
]
ASF GitHub Bot logged work on HIVE-26320:
-
Author: ASF GitHub Bot
Created on: 01/Oct/22 04:19
Start Date: 01/Oct/22 04:19
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3628:
URL: https://github.com/apache/hive/pull/3628#issuecomment-1264233942
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3628)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3628=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3628=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3628=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=CODE_SMELL)
[7 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3628=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3628=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 813832)
Time Spent: 4.5h (was: 4h 20m)
> Incorrect case evaluation for Parquet based table
> -
>
> Key: HIVE-26320
> URL: https://issues.apache.org/jira/browse/HIVE-26320
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 4.0.0-alpha-1
>Reporter: Chiran Ravani
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Query involving case statement with two or more conditions leads to incorrect
> result for tables with parquet format, The problem is not observed with ORC
> or TextFile.
> *Steps to reproduce*:
> {code:java}
> create external table case_test_parquet(kob varchar(2),enhanced_type_code
> int) stored as parquet;
> insert into case_test_parquet values('BB',18),('BC',18),('AB',18);
> select case when (
>(kob='BB' and enhanced_type_code='18')
>or (kob='BC' and enhanced_type_code='18')
> )
> then 1
> else 0
> end as logic_check
> from case_test_parquet;
> {code}
> Result:
> {code}
> 0
> 0
> 0
> {code}
> Expected result:
> {code}
> 1
> 1
> 0
> {code}
> The problem does not appear when setting hive.optimize.point.lookup=false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26254) upgrade calcite to 1.26 due to CVEs
[ https://issues.apache.org/jira/browse/HIVE-26254?focusedWorklogId=813824=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813824 ] ASF GitHub Bot logged work on HIVE-26254: - Author: ASF GitHub Bot Created on: 01/Oct/22 00:35 Start Date: 01/Oct/22 00:35 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on PR #3315: URL: https://github.com/apache/hive/pull/3315#issuecomment-1264166181 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. Issue Time Tracking --- Worklog Id: (was: 813824) Time Spent: 2h 40m (was: 2.5h) > upgrade calcite to 1.26 due to CVEs > --- > > Key: HIVE-26254 > URL: https://issues.apache.org/jira/browse/HIVE-26254 > Project: Hive > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > https://github.com/advisories/GHSA-hxp5-8pgq-mgv9 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?focusedWorklogId=813815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813815
]
ASF GitHub Bot logged work on HIVE-26320:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 23:30
Start Date: 30/Sep/22 23:30
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3628:
URL: https://github.com/apache/hive/pull/3628#issuecomment-1264127867
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3628)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3628=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3628=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3628=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=CODE_SMELL)
[7 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3628=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3628=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 813815)
Time Spent: 4h 20m (was: 4h 10m)
> Incorrect case evaluation for Parquet based table
> -
>
> Key: HIVE-26320
> URL: https://issues.apache.org/jira/browse/HIVE-26320
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 4.0.0-alpha-1
>Reporter: Chiran Ravani
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Query involving case statement with two or more conditions leads to incorrect
> result for tables with parquet format, The problem is not observed with ORC
> or TextFile.
> *Steps to reproduce*:
> {code:java}
> create external table case_test_parquet(kob varchar(2),enhanced_type_code
> int) stored as parquet;
> insert into case_test_parquet values('BB',18),('BC',18),('AB',18);
> select case when (
>(kob='BB' and enhanced_type_code='18')
>or (kob='BC' and enhanced_type_code='18')
> )
> then 1
> else 0
> end as logic_check
> from case_test_parquet;
> {code}
> Result:
> {code}
> 0
> 0
> 0
> {code}
> Expected result:
> {code}
> 1
> 1
> 0
> {code}
> The problem does not appear when setting hive.optimize.point.lookup=false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813808
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 22:56
Start Date: 30/Sep/22 22:56
Worklog Time Spent: 10m
Work Description: difin commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r984991177
##
ql/src/test/results/clientpositive/llap/subquery_ALL.q.out:
##
@@ -413,8 +413,7 @@ POSTHOOK: Input: default@part
POSTHOOK: Input: default@part_null_n0
A masked pattern was here
26
-Warning: Shuffle Join MERGEJOIN[37][tables = [$hdt$_1, $hdt$_2]] in Stage
'Reducer 3' is a cross product
-Warning: Shuffle Join MERGEJOIN[38][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in
Stage 'Reducer 4' is a cross product
+Warning: Shuffle Join MERGEJOIN[22][tables = [$hdt$_0, $hdt$_1]] in Stage
'Reducer 2' is a cross product
Review Comment:
Will these changes work for insert statements that produce zero records
also, or only for select queries?
I am asking because there is a ticket
https://issues.apache.org/jira/browse/HIVE-24466 - "insert queries should not
launch job when condition in the query would output 0 rows", I was wondering it
these changes are a fix for that ticket too.
Issue Time Tracking
---
Worklog Id: (was: 813808)
Time Spent: 5h 20m (was: 5h 10m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26166) Make website GDPR compliant
[ https://issues.apache.org/jira/browse/HIVE-26166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611770#comment-17611770 ] Simhadri Govindappa commented on HIVE-26166: Hi [~zabetak] , I would recommend waiting for HIVE-26565, because it replaces the entire hive-site. I will try to raise the PR for HIVE-26565 by end of the weekend. Thanks! > Make website GDPR compliant > --- > > Key: HIVE-26166 > URL: https://issues.apache.org/jira/browse/HIVE-26166 > Project: Hive > Issue Type: Task > Components: Website >Reporter: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Per the email that was sent out from privacy we need to make the Hive website > GDPR compliant. > # The link to privacy policy needs to be updated from > [https://hive.apache.org/privacy_policy.html] to > [https://privacy.apache.org/policies/privacy-policy-public.html] > # The google analytics service must be removed -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?focusedWorklogId=813801=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813801
]
ASF GitHub Bot logged work on HIVE-26320:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 21:06
Start Date: 30/Sep/22 21:06
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3628:
URL: https://github.com/apache/hive/pull/3628#issuecomment-1264024651
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3628)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3628=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3628=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3628=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=CODE_SMELL)
[7 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3628=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3628=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3628=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 813801)
Time Spent: 4h 10m (was: 4h)
> Incorrect case evaluation for Parquet based table
> -
>
> Key: HIVE-26320
> URL: https://issues.apache.org/jira/browse/HIVE-26320
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 4.0.0-alpha-1
>Reporter: Chiran Ravani
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Query involving case statement with two or more conditions leads to incorrect
> result for tables with parquet format, The problem is not observed with ORC
> or TextFile.
> *Steps to reproduce*:
> {code:java}
> create external table case_test_parquet(kob varchar(2),enhanced_type_code
> int) stored as parquet;
> insert into case_test_parquet values('BB',18),('BC',18),('AB',18);
> select case when (
>(kob='BB' and enhanced_type_code='18')
>or (kob='BC' and enhanced_type_code='18')
> )
> then 1
> else 0
> end as logic_check
> from case_test_parquet;
> {code}
> Result:
> {code}
> 0
> 0
> 0
> {code}
> Expected result:
> {code}
> 1
> 1
> 0
> {code}
> The problem does not appear when setting hive.optimize.point.lookup=false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26579) Prepare for Hadoop switching to Reload4j
[
https://issues.apache.org/jira/browse/HIVE-26579?focusedWorklogId=813789=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813789
]
ASF GitHub Bot logged work on HIVE-26579:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 18:40
Start Date: 30/Sep/22 18:40
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3635:
URL: https://github.com/apache/hive/pull/3635#issuecomment-1263898784
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3635)
[](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3635=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3635=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3635=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=CODE_SMELL)
[5 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3635=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3635=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 813789)
Time Spent: 0.5h (was: 20m)
> Prepare for Hadoop switching to Reload4j
>
>
> Key: HIVE-26579
> URL: https://issues.apache.org/jira/browse/HIVE-26579
> Project: Hive
> Issue Type: Task
> Components: Build Infrastructure
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task is
> to prepare Hive for that change:
> * Hive build fails with current {{useStrictFiltering=true}} setting in some
> assemblies, due to excluded dependency (log4j) not really being present.
> * Exclude {{ch.qos.reload4j:\*}} in addition to current {{log4j:\*}} to
> avoid polluting the assemblies and shaded jars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26576) Alter table calls on Iceberg tables can inadvertently change metadata_location
[ https://issues.apache.org/jira/browse/HIVE-26576?focusedWorklogId=813768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813768 ] ASF GitHub Bot logged work on HIVE-26576: - Author: ASF GitHub Bot Created on: 30/Sep/22 17:18 Start Date: 30/Sep/22 17:18 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3612: URL: https://github.com/apache/hive/pull/3612#issuecomment-1263824771 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3612) [](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=BUG) [5 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3612=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3612=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3612=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=CODE_SMELL) [11 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3612=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3612=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 813768) Time Spent: 1h (was: 50m) > Alter table calls on Iceberg tables can inadvertently change metadata_location > -- > > Key: HIVE-26576 > URL: https://issues.apache.org/jira/browse/HIVE-26576 > Project: Hive > Issue Type: Bug >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Concurrent alter_table calls can interfere and can cause the > metadata_location property of an Iceberg table to be messed up. > Basically there's no table level locking for Iceberg tables in Hive during > the usual operations, and thus some extra performance related features are > available, like concurrent inserts, as opposed to native Hive tables. This > was done under the assumption that the optimistic locking pattern that is > used in HiveTableOperations protects changing the metadata_location by the > use of an HMS table lock there only. > This is fine until some other alter_table calls get into the system such as > one from StatTask or DDLTask. Such tasks perform their work as: > * get the current table > * do the alteration > * send the changes via alter_table call to HMS > In between the retrieval of the table and the alter_table call a legit commit > from HiveTableOperations might bump the metadata_location, but this will get >
[jira] [Commented] (HIVE-26166) Make website GDPR compliant
[ https://issues.apache.org/jira/browse/HIVE-26166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611689#comment-17611689 ] Stamatis Zampetakis commented on HIVE-26166: [~simhadri-g] is working on revamping the Hive web-site in HIVE-26565. [~simhadri-g] Do you think we can merge [~martijnvisser] PR already or it is better to wait that HIVE-26565 goes in? > Make website GDPR compliant > --- > > Key: HIVE-26166 > URL: https://issues.apache.org/jira/browse/HIVE-26166 > Project: Hive > Issue Type: Task > Components: Website >Reporter: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Per the email that was sent out from privacy we need to make the Hive website > GDPR compliant. > # The link to privacy policy needs to be updated from > [https://hive.apache.org/privacy_policy.html] to > [https://privacy.apache.org/policies/privacy-policy-public.html] > # The google analytics service must be removed -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26166) Make website GDPR compliant
[ https://issues.apache.org/jira/browse/HIVE-26166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611687#comment-17611687 ] Stamatis Zampetakis commented on HIVE-26166: Thanks for the PRs [~martijnvisser]. I guess only the PR on the hive-site is required. I tried to build the site as well to test your changes before merging but it seems that the Dockerfile is broken. Will come back to this in the following week. > Make website GDPR compliant > --- > > Key: HIVE-26166 > URL: https://issues.apache.org/jira/browse/HIVE-26166 > Project: Hive > Issue Type: Task > Components: Website >Reporter: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Per the email that was sent out from privacy we need to make the Hive website > GDPR compliant. > # The link to privacy policy needs to be updated from > [https://hive.apache.org/privacy_policy.html] to > [https://privacy.apache.org/policies/privacy-policy-public.html] > # The google analytics service must be removed -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-13353) SHOW COMPACTIONS should support filtering options
[ https://issues.apache.org/jira/browse/HIVE-13353?focusedWorklogId=813758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813758 ] ASF GitHub Bot logged work on HIVE-13353: - Author: ASF GitHub Bot Created on: 30/Sep/22 16:58 Start Date: 30/Sep/22 16:58 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3608: URL: https://github.com/apache/hive/pull/3608#issuecomment-1263806616 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3608) [](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3608=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3608=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3608=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=CODE_SMELL) [11 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3608=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3608=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3608=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 813758) Time Spent: 50m (was: 40m) > SHOW COMPACTIONS should support filtering options > - > > Key: HIVE-13353 > URL: https://issues.apache.org/jira/browse/HIVE-13353 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.3.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: KIRTI RUGE >Priority: Major > Labels: pull-request-available > Attachments: HIVE-13353.01.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Since we now have historical information in SHOW COMPACTIONS the output can > easily become unwieldy. (e.g. 1000 partitions with 3 lines of history each) > this is a significant usability issue > Need to add ability to filter by db/table/partition > Perhaps would also be useful to filter by status -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26576) Alter table calls on Iceberg tables can inadvertently change metadata_location
[
https://issues.apache.org/jira/browse/HIVE-26576?focusedWorklogId=813751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813751
]
ASF GitHub Bot logged work on HIVE-26576:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 16:32
Start Date: 30/Sep/22 16:32
Worklog Time Spent: 10m
Work Description: szlta commented on code in PR #3612:
URL: https://github.com/apache/hive/pull/3612#discussion_r984767263
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##
@@ -386,6 +405,13 @@ private void
assertNotCrossTableMetadataLocationChange(Map tblPa
icebergTable.name(), newMetadataLocation)
);
}
+ if (!currentMetadata.metadataFileLocation().equals(newMetadataLocation)
&&
+
!context.getProperties().containsKey(MANUAL_ICEBERG_METADATA_LOCATION_CHANGE)) {
+// If we got here then this is an alter table operation where the
table to be changed had an Iceberg commit
+// meanwhile. The base metadata locations differ, while we know that
this wasn't an intentional, manual
+// metadata_location set by a user. To protect the interim commit we
need to refresh the metadata file location.
+tblParams.put(BaseMetastoreTableOperations.METADATA_LOCATION_PROP,
currentMetadata.metadataFileLocation());
Review Comment:
Good point! Fixed.
Issue Time Tracking
---
Worklog Id: (was: 813751)
Time Spent: 50m (was: 40m)
> Alter table calls on Iceberg tables can inadvertently change metadata_location
> --
>
> Key: HIVE-26576
> URL: https://issues.apache.org/jira/browse/HIVE-26576
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Concurrent alter_table calls can interfere and can cause the
> metadata_location property of an Iceberg table to be messed up.
> Basically there's no table level locking for Iceberg tables in Hive during
> the usual operations, and thus some extra performance related features are
> available, like concurrent inserts, as opposed to native Hive tables. This
> was done under the assumption that the optimistic locking pattern that is
> used in HiveTableOperations protects changing the metadata_location by the
> use of an HMS table lock there only.
> This is fine until some other alter_table calls get into the system such as
> one from StatTask or DDLTask. Such tasks perform their work as:
> * get the current table
> * do the alteration
> * send the changes via alter_table call to HMS
> In between the retrieval of the table and the alter_table call a legit commit
> from HiveTableOperations might bump the metadata_location, but this will get
> reverted as these tasks consider an outdated metadata_location (and the alter
> table call will overwrite all table props including this one too..)
> This is a design issue, and to solve this while preserving the concurrency
> features I propose to make use of HiveIcebergMetaHook where all such
> alter_table calls are intercepted, and the same locking mechanism could be
> used there as the one found in HiveTableOperations. The proposed flow on HMS
> client side would be:
> * hook: preAlterTable
> ** request table level lock
> ** refresh the Iceberg table from catalog (HMS) to see if new updates have
> arrived
> ** compare the current metadata with the one thought to be the base of this
> request, if metadata_location is outdated overwrite it with the fresh,
> current one in this request
> * do the alter_table call to HMS with the relevant changes (updated stats or
> other properties)
> * hook: post/rollbackAlterTable
> ** release table level lock
> This can work as the metadata_location should never be changed by anything
> other than HiveTableOperations, which is the only thing not using this hook
> (if it did we'd be in an endless loop). There's actually one exception which
> is if a user wants to change the metadata_location by hand. I can make an
> exception to that signalling this fact from an environmentContext instance
> when the corresponding AlterTableSetPropertiesDesc is constructed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?focusedWorklogId=813752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813752
]
ASF GitHub Bot logged work on HIVE-26320:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 16:32
Start Date: 30/Sep/22 16:32
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3628:
URL: https://github.com/apache/hive/pull/3628#discussion_r984763146
##
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java:
##
@@ -212,23 +216,61 @@ public void
testGetInt64NanosAdjustedToUTCTimestampConverter() throws Exception
}
@Test
- public void testGetTextConverter() throws Exception {
+ public void testGetTextConverterForString() throws Exception {
PrimitiveType primitiveType = Types.optional(PrimitiveTypeName.BINARY)
-.as(LogicalTypeAnnotation.stringType()).named("value");
-Writable writable = getWritableFromBinaryConverter(new VarcharTypeInfo(),
primitiveType,
-Binary.fromString("this_is_a_value"));
-Text textWritable = (Text) writable;
-assertEquals("this_is_a_value", textWritable.toString());
+.as(LogicalTypeAnnotation.stringType()).named("value");
+String value = "this_is_a_value";
+Text textWritable = (Text) getWritableFromBinaryConverter(stringTypeInfo,
primitiveType,
+Binary.fromString(value));
+assertEquals(value, textWritable.toString());
}
@Test
- public void testGetTextConverterNoHiveTypeInfo() throws Exception {
+ public void testGetTextConverterForCharPadsValueWithSpacesTillLen() {
+PrimitiveType primitiveType = Types.optional(PrimitiveTypeName.BINARY)
+.as(LogicalTypeAnnotation.stringType()).named("value");
+String value = "this_is_a_value";
+HiveCharWritable textWritable = (HiveCharWritable)
getWritableFromBinaryConverter(
+new CharTypeInfo(value.length() + 2), primitiveType,
Binary.fromString(value));
+assertEquals(value + " ", textWritable.toString());
+ }
+
+ public void testGetTextConverterForCharTruncatesValueExceedingLen() {
Review Comment:
Probably the `@Test` annotations needs to be added here.
Issue Time Tracking
---
Worklog Id: (was: 813752)
Time Spent: 4h (was: 3h 50m)
> Incorrect case evaluation for Parquet based table
> -
>
> Key: HIVE-26320
> URL: https://issues.apache.org/jira/browse/HIVE-26320
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 4.0.0-alpha-1
>Reporter: Chiran Ravani
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Query involving case statement with two or more conditions leads to incorrect
> result for tables with parquet format, The problem is not observed with ORC
> or TextFile.
> *Steps to reproduce*:
> {code:java}
> create external table case_test_parquet(kob varchar(2),enhanced_type_code
> int) stored as parquet;
> insert into case_test_parquet values('BB',18),('BC',18),('AB',18);
> select case when (
>(kob='BB' and enhanced_type_code='18')
>or (kob='BC' and enhanced_type_code='18')
> )
> then 1
> else 0
> end as logic_check
> from case_test_parquet;
> {code}
> Result:
> {code}
> 0
> 0
> 0
> {code}
> Expected result:
> {code}
> 1
> 1
> 0
> {code}
> The problem does not appear when setting hive.optimize.point.lookup=false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26583) Ensure iceberg-catalog tests are executed in ptest
[
https://issues.apache.org/jira/browse/HIVE-26583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita reassigned HIVE-26583:
-
> Ensure iceberg-catalog tests are executed in ptest
> --
>
> Key: HIVE-26583
> URL: https://issues.apache.org/jira/browse/HIVE-26583
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
>
> When running Iceberg tests locally I discovered that there's a failing case
> in iceberg-catalog
> {{HiveCreateReplaceTableTest.testReplaceTableTxnTableNotExists}}, failing
> with
> {code}
> java.lang.AssertionError: Expected exception message (No such table:
> hivedb.tbl) missing: Table does not exist: hivedb.tbl
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at
> org.apache.iceberg.AssertHelpers.handleException(AssertHelpers.java:129)
> at org.apache.iceberg.AssertHelpers.assertThrows(AssertHelpers.java:47)
> at
> org.apache.iceberg.hive.HiveCreateReplaceTableTest.testReplaceTableTxnTableNotExists(HiveCreateReplaceTableTest.java:168)
> {code}
> and it probably has been like that since one of the recent Iceberg dependency
> upgrades.
> We should fix this test so that it has the expected type of exception and
> ensure that this module is verified as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?focusedWorklogId=813742=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813742
]
ASF GitHub Bot logged work on HIVE-26320:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 15:33
Start Date: 30/Sep/22 15:33
Worklog Time Spent: 10m
Work Description: jfsii commented on code in PR #3628:
URL: https://github.com/apache/hive/pull/3628#discussion_r984716423
##
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveBaseCharWritable.java:
##
@@ -26,10 +26,19 @@
import org.apache.hive.common.util.HiveStringUtils;
public abstract class HiveBaseCharWritable {
- protected Text value = new Text();
+ protected Text value;
protected int charLength = -1;
public HiveBaseCharWritable() {
+value = new Text();
+ }
+
+ public HiveBaseCharWritable(Text text) {
+value = text;
+ }
+
+ public HiveBaseCharWritable(byte[] bytes) {
+value = new Text(bytes);
Review Comment:
Likely no specific reason, it just happened to end up that way. I suppose
that I didn't notice the set method on the Text object and wanted to avoid an
extra allocation.
Issue Time Tracking
---
Worklog Id: (was: 813742)
Time Spent: 3h 50m (was: 3h 40m)
> Incorrect case evaluation for Parquet based table
> -
>
> Key: HIVE-26320
> URL: https://issues.apache.org/jira/browse/HIVE-26320
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 4.0.0-alpha-1
>Reporter: Chiran Ravani
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Query involving case statement with two or more conditions leads to incorrect
> result for tables with parquet format, The problem is not observed with ORC
> or TextFile.
> *Steps to reproduce*:
> {code:java}
> create external table case_test_parquet(kob varchar(2),enhanced_type_code
> int) stored as parquet;
> insert into case_test_parquet values('BB',18),('BC',18),('AB',18);
> select case when (
>(kob='BB' and enhanced_type_code='18')
>or (kob='BC' and enhanced_type_code='18')
> )
> then 1
> else 0
> end as logic_check
> from case_test_parquet;
> {code}
> Result:
> {code}
> 0
> 0
> 0
> {code}
> Expected result:
> {code}
> 1
> 1
> 0
> {code}
> The problem does not appear when setting hive.optimize.point.lookup=false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?focusedWorklogId=813741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813741
]
ASF GitHub Bot logged work on HIVE-26320:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 15:32
Start Date: 30/Sep/22 15:32
Worklog Time Spent: 10m
Work Description: jfsii commented on code in PR #3628:
URL: https://github.com/apache/hive/pull/3628#discussion_r984714917
##
ql/src/test/queries/clientpositive/pointlookup.q:
##
@@ -127,3 +127,39 @@ or inOutputOpt.key = null;
drop table orOutput;
drop table inOutput;
drop table inOutputOpt;
+
+
Issue Time Tracking
---
Worklog Id: (was: 813741)
Time Spent: 3h 40m (was: 3.5h)
> Incorrect case evaluation for Parquet based table
> -
>
> Key: HIVE-26320
> URL: https://issues.apache.org/jira/browse/HIVE-26320
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 4.0.0-alpha-1
>Reporter: Chiran Ravani
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Query involving case statement with two or more conditions leads to incorrect
> result for tables with parquet format, The problem is not observed with ORC
> or TextFile.
> *Steps to reproduce*:
> {code:java}
> create external table case_test_parquet(kob varchar(2),enhanced_type_code
> int) stored as parquet;
> insert into case_test_parquet values('BB',18),('BC',18),('AB',18);
> select case when (
>(kob='BB' and enhanced_type_code='18')
>or (kob='BC' and enhanced_type_code='18')
> )
> then 1
> else 0
> end as logic_check
> from case_test_parquet;
> {code}
> Result:
> {code}
> 0
> 0
> 0
> {code}
> Expected result:
> {code}
> 1
> 1
> 0
> {code}
> The problem does not appear when setting hive.optimize.point.lookup=false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?focusedWorklogId=813740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813740
]
ASF GitHub Bot logged work on HIVE-26320:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 15:31
Start Date: 30/Sep/22 15:31
Worklog Time Spent: 10m
Work Description: jfsii commented on code in PR #3628:
URL: https://github.com/apache/hive/pull/3628#discussion_r984714606
##
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java:
##
@@ -481,12 +485,36 @@ protected BytesWritable convert(Binary binary) {
},
ESTRING_CONVERTER(String.class) {
@Override
-PrimitiveConverter getConverter(final PrimitiveType type, final int index,
final ConverterParent parent, TypeInfo hiveTypeInfo) {
+PrimitiveConverter getConverter(final PrimitiveType type, final int index,
final ConverterParent parent,
+TypeInfo hiveTypeInfo) {
+ // If we have type information, we should return properly typed strings.
However, there are a variety
+ // of code paths that do not provide the typeInfo in those cases we
default to Text. This idiom is also
+ // followed by for example the BigDecimal converter in which if there is
no type information,
+ // it defaults to the widest representation
+ if (hiveTypeInfo != null) {
+String typeName = hiveTypeInfo.getTypeName().toLowerCase();
+if (typeName.startsWith(serdeConstants.CHAR_TYPE_NAME)) {
+ return new BinaryConverter(type, parent, index) {
+@Override
+ protected HiveCharWritable convert(Binary binary) {
+return new HiveCharWritable(binary.getBytes(), ((CharTypeInfo)
hiveTypeInfo).getLength());
+ }
+ };
+} else if (typeName.startsWith(serdeConstants.VARCHAR_TYPE_NAME)) {
+ return new BinaryConverter(type, parent, index)
{
+@Override
+ protected HiveVarcharWritable convert(Binary binary) {
+return new HiveVarcharWritable(binary.getBytes(),
((VarcharTypeInfo) hiveTypeInfo).getLength());
+ }
+ };
+}
+ }
Review Comment:
Yeah, I did follow the file on its convention. However, I changed it to your
suggestion because it looks cleaner and might as well encourage this still in
case someone copies me
Issue Time Tracking
---
Worklog Id: (was: 813740)
Time Spent: 3.5h (was: 3h 20m)
> Incorrect case evaluation for Parquet based table
> -
>
> Key: HIVE-26320
> URL: https://issues.apache.org/jira/browse/HIVE-26320
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 4.0.0-alpha-1
>Reporter: Chiran Ravani
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Query involving case statement with two or more conditions leads to incorrect
> result for tables with parquet format, The problem is not observed with ORC
> or TextFile.
> *Steps to reproduce*:
> {code:java}
> create external table case_test_parquet(kob varchar(2),enhanced_type_code
> int) stored as parquet;
> insert into case_test_parquet values('BB',18),('BC',18),('AB',18);
> select case when (
>(kob='BB' and enhanced_type_code='18')
>or (kob='BC' and enhanced_type_code='18')
> )
> then 1
> else 0
> end as logic_check
> from case_test_parquet;
> {code}
> Result:
> {code}
> 0
> 0
> 0
> {code}
> Expected result:
> {code}
> 1
> 1
> 0
> {code}
> The problem does not appear when setting hive.optimize.point.lookup=false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26582) Cartesian join fails if the query has an empty table when cartesian product edge is used
[
https://issues.apache.org/jira/browse/HIVE-26582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sourabh Badhya updated HIVE-26582:
--
Description:
The following example fails when "hive.tez.cartesian-product.enabled" is
enabled -
Test command -
{code:java}
mvn test -Dtest=TestMiniLlapCliDriver -Dqfile=file.q
-Dtest.output.overwrite=true {code}
Query - file.q
{code:java}
set hive.tez.cartesian-product.enabled=true;
create table c (a1 int) stored as orc;
create table tmp1 (a int) stored as orc;
create table tmp2 (a int) stored as orc;
insert into table c values (3);
insert into table tmp1 values (3);
with
first as (
select a1 from c where a1 = 3
),
second as (
select a from tmp1
union all
select a from tmp2
)
select a from second cross join first; {code}
The following stack trace is seen -
{code:java}
Caused by: java.lang.IllegalArgumentException: Number of items is 0. Should be
positive
at
org.apache.tez.common.Preconditions.checkArgument(Preconditions.java:38)
at org.apache.tez.runtime.library.utils.Grouper.init(Grouper.java:41)
at
org.apache.tez.runtime.library.cartesianproduct.FairCartesianProductEdgeManager.initialize(FairCartesianProductEdgeManager.java:66)
at
org.apache.tez.runtime.library.cartesianproduct.CartesianProductEdgeManager.initialize(CartesianProductEdgeManager.java:51)
at org.apache.tez.dag.app.dag.impl.Edge.initialize(Edge.java:213)
... 22 more{code}
The following error is seen because one of the tables (tmp2 in this case) has 0
rows in it.
The query works fine when the config hive.tez.cartesian-product.enabled is set
to false.
was:
The following example fails when "hive.tez.cartesian-product.enabled" is
enabled -
Test command -
{code:java}
mvn test -Dtest=TestMiniLlapCliDriver -Dqfile=file.q
-Dtest.output.overwrite=true {code}
Query - file.q
{code:java}
set hive.tez.cartesian-product.enabled=true;
create table c (a1 int) stored as orc;
create table tmp1 (a int) stored as orc;
create table tmp2 (a int) stored as orc;
insert into table c values (3);
insert into table tmp1 values (3);
with
first as (
select a1 from c where a1 = 3
),
second as (
select a from tmp1
union all
select a from tmp2
)
select a from second cross join first; {code}
The following stack trace is seen -
{code:java}
Caused by: java.lang.IllegalArgumentException: Number of items is 0. Should be
positive
at
org.apache.tez.common.Preconditions.checkArgument(Preconditions.java:38)
at org.apache.tez.runtime.library.utils.Grouper.init(Grouper.java:41)
at
org.apache.tez.runtime.library.cartesianproduct.FairCartesianProductEdgeManager.initialize(FairCartesianProductEdgeManager.java:66)
at
org.apache.tez.runtime.library.cartesianproduct.CartesianProductEdgeManager.initialize(CartesianProductEdgeManager.java:51)
at org.apache.tez.dag.app.dag.impl.Edge.initialize(Edge.java:213)
... 22 more{code}
The following error is seen because one of the tables (tmp2 in this case) has 0
rows in it.
The query works fine when the config hive.tez.cartesian-product.enabled is set
to false.
> Cartesian join fails if the query has an empty table when cartesian product
> edge is used
>
>
> Key: HIVE-26582
> URL: https://issues.apache.org/jira/browse/HIVE-26582
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Priority: Major
>
> The following example fails when "hive.tez.cartesian-product.enabled" is
> enabled -
> Test command -
> {code:java}
> mvn test -Dtest=TestMiniLlapCliDriver -Dqfile=file.q
> -Dtest.output.overwrite=true {code}
> Query - file.q
> {code:java}
> set hive.tez.cartesian-product.enabled=true;
> create table c (a1 int) stored as orc;
> create table tmp1 (a int) stored as orc;
> create table tmp2 (a int) stored as orc;
> insert into table c values (3);
> insert into table tmp1 values (3);
> with
> first as (
> select a1 from c where a1 = 3
> ),
> second as (
> select a from tmp1
> union all
> select a from tmp2
> )
> select a from second cross join first; {code}
> The following stack trace is seen -
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Number of items is 0. Should
> be positive
> at
> org.apache.tez.common.Preconditions.checkArgument(Preconditions.java:38)
> at org.apache.tez.runtime.library.utils.Grouper.init(Grouper.java:41)
> at
> org.apache.tez.runtime.library.cartesianproduct.FairCartesianProductEdgeManager.initialize(FairCartesianProductEdgeManager.java:66)
> at
> org.apache.tez.runtime.library.cartesianproduct.CartesianProductEdgeManager.initialize(CartesianProductEdgeManager.java:51)
> at org.apache.tez.dag.app.dag.impl.Edge.initialize(Edge.java:213)
> ...
[jira] [Updated] (HIVE-26582) Cartesian join fails if the query has an empty table when cartesian product edge is used
[
https://issues.apache.org/jira/browse/HIVE-26582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sourabh Badhya updated HIVE-26582:
--
Description:
The following example fails when "hive.tez.cartesian-product.enabled" is true -
Test command -
{code:java}
mvn test -Dtest=TestMiniLlapCliDriver -Dqfile=file.q
-Dtest.output.overwrite=true {code}
Query - file.q
{code:java}
set hive.tez.cartesian-product.enabled=true;
create table c (a1 int) stored as orc;
create table tmp1 (a int) stored as orc;
create table tmp2 (a int) stored as orc;
insert into table c values (3);
insert into table tmp1 values (3);
with
first as (
select a1 from c where a1 = 3
),
second as (
select a from tmp1
union all
select a from tmp2
)
select a from second cross join first; {code}
The following stack trace is seen -
{code:java}
Caused by: java.lang.IllegalArgumentException: Number of items is 0. Should be
positive
at
org.apache.tez.common.Preconditions.checkArgument(Preconditions.java:38)
at org.apache.tez.runtime.library.utils.Grouper.init(Grouper.java:41)
at
org.apache.tez.runtime.library.cartesianproduct.FairCartesianProductEdgeManager.initialize(FairCartesianProductEdgeManager.java:66)
at
org.apache.tez.runtime.library.cartesianproduct.CartesianProductEdgeManager.initialize(CartesianProductEdgeManager.java:51)
at org.apache.tez.dag.app.dag.impl.Edge.initialize(Edge.java:213)
... 22 more{code}
The following error is seen because one of the tables (tmp2 in this case) has 0
rows in it.
The query works fine when the config hive.tez.cartesian-product.enabled is set
to false.
was:
The following example fails when "hive.tez.cartesian-product.enabled" is
enabled -
Test command -
{code:java}
mvn test -Dtest=TestMiniLlapCliDriver -Dqfile=file.q
-Dtest.output.overwrite=true {code}
Query - file.q
{code:java}
set hive.tez.cartesian-product.enabled=true;
create table c (a1 int) stored as orc;
create table tmp1 (a int) stored as orc;
create table tmp2 (a int) stored as orc;
insert into table c values (3);
insert into table tmp1 values (3);
with
first as (
select a1 from c where a1 = 3
),
second as (
select a from tmp1
union all
select a from tmp2
)
select a from second cross join first; {code}
The following stack trace is seen -
{code:java}
Caused by: java.lang.IllegalArgumentException: Number of items is 0. Should be
positive
at
org.apache.tez.common.Preconditions.checkArgument(Preconditions.java:38)
at org.apache.tez.runtime.library.utils.Grouper.init(Grouper.java:41)
at
org.apache.tez.runtime.library.cartesianproduct.FairCartesianProductEdgeManager.initialize(FairCartesianProductEdgeManager.java:66)
at
org.apache.tez.runtime.library.cartesianproduct.CartesianProductEdgeManager.initialize(CartesianProductEdgeManager.java:51)
at org.apache.tez.dag.app.dag.impl.Edge.initialize(Edge.java:213)
... 22 more{code}
The following error is seen because one of the tables (tmp2 in this case) has 0
rows in it.
The query works fine when the config hive.tez.cartesian-product.enabled is set
to false.
> Cartesian join fails if the query has an empty table when cartesian product
> edge is used
>
>
> Key: HIVE-26582
> URL: https://issues.apache.org/jira/browse/HIVE-26582
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Priority: Major
>
> The following example fails when "hive.tez.cartesian-product.enabled" is true
> -
> Test command -
> {code:java}
> mvn test -Dtest=TestMiniLlapCliDriver -Dqfile=file.q
> -Dtest.output.overwrite=true {code}
> Query - file.q
> {code:java}
> set hive.tez.cartesian-product.enabled=true;
> create table c (a1 int) stored as orc;
> create table tmp1 (a int) stored as orc;
> create table tmp2 (a int) stored as orc;
> insert into table c values (3);
> insert into table tmp1 values (3);
> with
> first as (
> select a1 from c where a1 = 3
> ),
> second as (
> select a from tmp1
> union all
> select a from tmp2
> )
> select a from second cross join first; {code}
> The following stack trace is seen -
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Number of items is 0. Should
> be positive
> at
> org.apache.tez.common.Preconditions.checkArgument(Preconditions.java:38)
> at org.apache.tez.runtime.library.utils.Grouper.init(Grouper.java:41)
> at
> org.apache.tez.runtime.library.cartesianproduct.FairCartesianProductEdgeManager.initialize(FairCartesianProductEdgeManager.java:66)
> at
> org.apache.tez.runtime.library.cartesianproduct.CartesianProductEdgeManager.initialize(CartesianProductEdgeManager.java:51)
> at org.apache.tez.dag.app.dag.impl.Edge.initialize(Edge.java:213)
> ... 22 more{code}
> The
[jira] [Updated] (HIVE-26582) Cartesian join fails if the query has an empty table when cartesian product edge is used
[
https://issues.apache.org/jira/browse/HIVE-26582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sourabh Badhya updated HIVE-26582:
--
Description:
The following example fails when "hive.tez.cartesian-product.enabled" is
enabled -
Test command -
{code:java}
mvn test -Dtest=TestMiniLlapCliDriver -Dqfile=file.q
-Dtest.output.overwrite=true {code}
Query - file.q
{code:java}
set hive.tez.cartesian-product.enabled=true;
create table c (a1 int) stored as orc;
create table tmp1 (a int) stored as orc;
create table tmp2 (a int) stored as orc;
insert into table c values (3);
insert into table tmp1 values (3);
with
first as (
select a1 from c where a1 = 3
),
second as (
select a from tmp1
union all
select a from tmp2
)
select a from second cross join first; {code}
The following stack trace is seen -
{code:java}
Caused by: java.lang.IllegalArgumentException: Number of items is 0. Should be
positive
at
org.apache.tez.common.Preconditions.checkArgument(Preconditions.java:38)
at org.apache.tez.runtime.library.utils.Grouper.init(Grouper.java:41)
at
org.apache.tez.runtime.library.cartesianproduct.FairCartesianProductEdgeManager.initialize(FairCartesianProductEdgeManager.java:66)
at
org.apache.tez.runtime.library.cartesianproduct.CartesianProductEdgeManager.initialize(CartesianProductEdgeManager.java:51)
at org.apache.tez.dag.app.dag.impl.Edge.initialize(Edge.java:213)
... 22 more{code}
The following error is seen because one of the tables (tmp2 in this case) has 0
rows in it.
The query works fine when the config hive.tez.cartesian-product.enabled is set
to false.
was:
The following example fails when "hive.tez.cartesian-product.enabled" is
enabled -
Test command -
{code:java}
mvn test -Dtest=TestMiniLlapCliDriver -Dqfile=file.q
-Dtest.output.overwrite=true {code}
Query - file.q
{code:java}
set hive.tez.cartesian-product.enabled=true;
create table c (a1 int) stored as orc;
create table tmp1 (a int) stored as orc;
create table tmp2 (a int) stored as orc;
insert into table c values (3);
insert into table tmp1 values (3);
with
first as (
select a1 from c where a1 = 3
),
second as (
select a from tmp1
union all
select a from tmp2
)
select a from second cross join first; {code}
The following stack trace is seen -
{code:java}
Caused by: java.lang.IllegalArgumentException: Number of items is 0. Should be
positive
at
org.apache.tez.common.Preconditions.checkArgument(Preconditions.java:38)
at org.apache.tez.runtime.library.utils.Grouper.init(Grouper.java:41)
at
org.apache.tez.runtime.library.cartesianproduct.FairCartesianProductEdgeManager.initialize(FairCartesianProductEdgeManager.java:66)
at
org.apache.tez.runtime.library.cartesianproduct.CartesianProductEdgeManager.initialize(CartesianProductEdgeManager.java:51)
at org.apache.tez.dag.app.dag.impl.Edge.initialize(Edge.java:213)
... 22 more{code}
The following error is seen because one of the tables (tmp2 in this case) has 0
rows in it.
The query works fine when the config hive.tez.cartesian-product.enabled is set
to false.
> Cartesian join fails if the query has an empty table when cartesian product
> edge is used
>
>
> Key: HIVE-26582
> URL: https://issues.apache.org/jira/browse/HIVE-26582
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Badhya
>Priority: Major
>
> The following example fails when "hive.tez.cartesian-product.enabled" is
> enabled -
> Test command -
> {code:java}
> mvn test -Dtest=TestMiniLlapCliDriver -Dqfile=file.q
> -Dtest.output.overwrite=true {code}
> Query - file.q
> {code:java}
> set hive.tez.cartesian-product.enabled=true;
> create table c (a1 int) stored as orc;
> create table tmp1 (a int) stored as orc;
> create table tmp2 (a int) stored as orc;
> insert into table c values (3);
> insert into table tmp1 values (3);
> with
> first as (
> select a1 from c where a1 = 3
> ),
> second as (
> select a from tmp1
> union all
> select a from tmp2
> )
> select a from second cross join first; {code}
> The following stack trace is seen -
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Number of items is 0. Should
> be positive
> at
> org.apache.tez.common.Preconditions.checkArgument(Preconditions.java:38)
> at org.apache.tez.runtime.library.utils.Grouper.init(Grouper.java:41)
> at
> org.apache.tez.runtime.library.cartesianproduct.FairCartesianProductEdgeManager.initialize(FairCartesianProductEdgeManager.java:66)
> at
> org.apache.tez.runtime.library.cartesianproduct.CartesianProductEdgeManager.initialize(CartesianProductEdgeManager.java:51)
> at org.apache.tez.dag.app.dag.impl.Edge.initialize(Edge.java:213)
> ... 22
[jira] [Updated] (HIVE-26582) Cartesian join fails if the query has an empty table when cartesian product edge is used
[
https://issues.apache.org/jira/browse/HIVE-26582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sourabh Badhya updated HIVE-26582:
--
Component/s: Hive
Tez
> Cartesian join fails if the query has an empty table when cartesian product
> edge is used
>
>
> Key: HIVE-26582
> URL: https://issues.apache.org/jira/browse/HIVE-26582
> Project: Hive
> Issue Type: Bug
> Components: Hive, Tez
>Reporter: Sourabh Badhya
>Priority: Major
>
> The following example fails when "hive.tez.cartesian-product.enabled" is true
> -
> Test command -
> {code:java}
> mvn test -Dtest=TestMiniLlapCliDriver -Dqfile=file.q
> -Dtest.output.overwrite=true {code}
> Query - file.q
> {code:java}
> set hive.tez.cartesian-product.enabled=true;
> create table c (a1 int) stored as orc;
> create table tmp1 (a int) stored as orc;
> create table tmp2 (a int) stored as orc;
> insert into table c values (3);
> insert into table tmp1 values (3);
> with
> first as (
> select a1 from c where a1 = 3
> ),
> second as (
> select a from tmp1
> union all
> select a from tmp2
> )
> select a from second cross join first; {code}
> The following stack trace is seen -
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Number of items is 0. Should
> be positive
> at
> org.apache.tez.common.Preconditions.checkArgument(Preconditions.java:38)
> at org.apache.tez.runtime.library.utils.Grouper.init(Grouper.java:41)
> at
> org.apache.tez.runtime.library.cartesianproduct.FairCartesianProductEdgeManager.initialize(FairCartesianProductEdgeManager.java:66)
> at
> org.apache.tez.runtime.library.cartesianproduct.CartesianProductEdgeManager.initialize(CartesianProductEdgeManager.java:51)
> at org.apache.tez.dag.app.dag.impl.Edge.initialize(Edge.java:213)
> ... 22 more{code}
> The following error is seen because one of the tables (tmp2 in this case) has
> 0 rows in it.
> The query works fine when the config hive.tez.cartesian-product.enabled is
> set to false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26579) Prepare for Hadoop switching to Reload4j
[
https://issues.apache.org/jira/browse/HIVE-26579?focusedWorklogId=813702=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813702
]
ASF GitHub Bot logged work on HIVE-26579:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 13:55
Start Date: 30/Sep/22 13:55
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3635:
URL: https://github.com/apache/hive/pull/3635#issuecomment-1263608541
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3635)
[](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3635=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3635=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3635=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=CODE_SMELL)
[5 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3635=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3635=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3635=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 813702)
Time Spent: 20m (was: 10m)
> Prepare for Hadoop switching to Reload4j
>
>
> Key: HIVE-26579
> URL: https://issues.apache.org/jira/browse/HIVE-26579
> Project: Hive
> Issue Type: Task
> Components: Build Infrastructure
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task is
> to prepare Hive for that change:
> * Hive build fails with current {{useStrictFiltering=true}} setting in some
> assemblies, due to excluded dependency (log4j) not really being present.
> * Exclude {{ch.qos.reload4j:\*}} in addition to current {{log4j:\*}} to
> avoid polluting the assemblies and shaded jars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-26579) Prepare for Hadoop switching to Reload4j
[
https://issues.apache.org/jira/browse/HIVE-26579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-26579 started by Attila Doroszlai.
---
> Prepare for Hadoop switching to Reload4j
>
>
> Key: HIVE-26579
> URL: https://issues.apache.org/jira/browse/HIVE-26579
> Project: Hive
> Issue Type: Task
> Components: Build Infrastructure
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task is
> to prepare Hive for that change:
> * Hive build fails with current {{useStrictFiltering=true}} setting in some
> assemblies, due to excluded dependency (log4j) not really being present.
> * Exclude {{ch.qos.reload4j:\*}} in addition to current {{log4j:\*}} to
> avoid polluting the assemblies and shaded jars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26579) Prepare for Hadoop switching to Reload4j
[
https://issues.apache.org/jira/browse/HIVE-26579?focusedWorklogId=813686=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813686
]
ASF GitHub Bot logged work on HIVE-26579:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 13:11
Start Date: 30/Sep/22 13:11
Worklog Time Spent: 10m
Work Description: adoroszlai opened a new pull request, #3635:
URL: https://github.com/apache/hive/pull/3635
## What changes were proposed in this pull request?
Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task
is to prepare Hive for that change:
* Hive build fails with current `useStrictFiltering=true` setting in some
assemblies, due to excluded dependency (log4j) not really being present.
* Exclude `ch.qos.reload4j:*` in addition to current `log4j:*` to avoid
polluting the assemblies and shaded jars.
https://issues.apache.org/jira/browse/HIVE-26579
## How was this patch tested?
CI build was successful:
https://github.com/adoroszlai/hive/actions/runs/3158692928/jobs/5141014133
Built Hive locally, checked artifacts affected by the change still do not
have Log4j1 jar/classes:
```
$ mvn -DskipTests -Pdist clean install
...
[INFO] BUILD SUCCESS
$ tar tzvf
standalone-metastore/metastore-server/target/apache-hive-standalone-metastore-server-4.0.0-alpha-2-SNAPSHOT-bin.tar.gz
| grep log4j
-rw-r--r
Issue Time Tracking
---
Worklog Id: (was: 813686)
Remaining Estimate: 0h
Time Spent: 10m
> Prepare for Hadoop switching to Reload4j
>
>
> Key: HIVE-26579
> URL: https://issues.apache.org/jira/browse/HIVE-26579
> Project: Hive
> Issue Type: Task
> Components: Build Infrastructure
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task is
> to prepare Hive for that change:
> * Hive build fails with current {{useStrictFiltering=true}} setting in some
> assemblies, due to excluded dependency (log4j) not really being present.
> * Exclude {{ch.qos.reload4j:\*}} in addition to current {{log4j:\*}} to
> avoid polluting the assemblies and shaded jars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26579) Prepare for Hadoop switching to Reload4j
[
https://issues.apache.org/jira/browse/HIVE-26579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26579:
--
Labels: pull-request-available (was: )
> Prepare for Hadoop switching to Reload4j
>
>
> Key: HIVE-26579
> URL: https://issues.apache.org/jira/browse/HIVE-26579
> Project: Hive
> Issue Type: Task
> Components: Build Infrastructure
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task is
> to prepare Hive for that change:
> * Hive build fails with current {{useStrictFiltering=true}} setting in some
> assemblies, due to excluded dependency (log4j) not really being present.
> * Exclude {{ch.qos.reload4j:\*}} in addition to current {{log4j:\*}} to
> avoid polluting the assemblies and shaded jars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[ https://issues.apache.org/jira/browse/HIVE-26580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KIRTI RUGE reassigned HIVE-26580: - Assignee: KIRTI RUGE > SHOW COMPACTIONS should support ordering and limiting functionality in > filtering options > > > Key: HIVE-26580 > URL: https://issues.apache.org/jira/browse/HIVE-26580 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: KIRTI RUGE >Assignee: KIRTI RUGE >Priority: Major > Fix For: 4.0.0 > > > SHOW COMPACTION should provide ordering by defied table . It should also > support limitation of fetched records -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26579) Prepare for Hadoop switching to Reload4j
[
https://issues.apache.org/jira/browse/HIVE-26579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai reassigned HIVE-26579:
---
> Prepare for Hadoop switching to Reload4j
>
>
> Key: HIVE-26579
> URL: https://issues.apache.org/jira/browse/HIVE-26579
> Project: Hive
> Issue Type: Task
> Components: Build Infrastructure
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
>
> Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task is
> to prepare Hive for that change:
> * Hive build fails with current {{useStrictFiltering=true}} setting in some
> assemblies, due to excluded dependency (log4j) not really being present.
> * Exclude {{ch.qos.reload4j:*}} in addition to current {{log4j:*}} to avoid
> polluting the assemblies and shaded jars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26579) Prepare for Hadoop switching to Reload4j
[
https://issues.apache.org/jira/browse/HIVE-26579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HIVE-26579:
Description:
Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task is
to prepare Hive for that change:
* Hive build fails with current {{useStrictFiltering=true}} setting in some
assemblies, due to excluded dependency (log4j) not really being present.
* Exclude {{ch.qos.reload4j:\*}} in addition to current {{log4j:*}} to avoid
polluting the assemblies and shaded jars.
was:
Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task is
to prepare Hive for that change:
* Hive build fails with current {{useStrictFiltering=true}} setting in some
assemblies, due to excluded dependency (log4j) not really being present.
* Exclude {{ch.qos.reload4j:*}} in addition to current {{log4j:*}} to avoid
polluting the assemblies and shaded jars.
> Prepare for Hadoop switching to Reload4j
>
>
> Key: HIVE-26579
> URL: https://issues.apache.org/jira/browse/HIVE-26579
> Project: Hive
> Issue Type: Task
> Components: Build Infrastructure
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
>
> Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task is
> to prepare Hive for that change:
> * Hive build fails with current {{useStrictFiltering=true}} setting in some
> assemblies, due to excluded dependency (log4j) not really being present.
> * Exclude {{ch.qos.reload4j:\*}} in addition to current {{log4j:*}} to avoid
> polluting the assemblies and shaded jars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26579) Prepare for Hadoop switching to Reload4j
[
https://issues.apache.org/jira/browse/HIVE-26579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HIVE-26579:
Description:
Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task is
to prepare Hive for that change:
* Hive build fails with current {{useStrictFiltering=true}} setting in some
assemblies, due to excluded dependency (log4j) not really being present.
* Exclude {{ch.qos.reload4j:\*}} in addition to current {{log4j:\*}} to avoid
polluting the assemblies and shaded jars.
was:
Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task is
to prepare Hive for that change:
* Hive build fails with current {{useStrictFiltering=true}} setting in some
assemblies, due to excluded dependency (log4j) not really being present.
* Exclude {{ch.qos.reload4j:\*}} in addition to current {{log4j:*}} to avoid
polluting the assemblies and shaded jars.
> Prepare for Hadoop switching to Reload4j
>
>
> Key: HIVE-26579
> URL: https://issues.apache.org/jira/browse/HIVE-26579
> Project: Hive
> Issue Type: Task
> Components: Build Infrastructure
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
>
> Hadoop moved from Log4j1 to Reload4j (HADOOP-18088). The goal of this task is
> to prepare Hive for that change:
> * Hive build fails with current {{useStrictFiltering=true}} setting in some
> assemblies, due to excluded dependency (log4j) not really being present.
> * Exclude {{ch.qos.reload4j:\*}} in addition to current {{log4j:\*}} to
> avoid polluting the assemblies and shaded jars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813666
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 11:11
Start Date: 30/Sep/22 11:11
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r984481438
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java:
##
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.core.Union;
+import org.apache.calcite.rel.core.Values;
+import org.apache.calcite.rel.rules.PruneEmptyRules;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.hadoop.hive.ql.optimizer.calcite.Bug;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+
+import java.util.Collections;
+import java.util.List;
+
+import static com.google.common.collect.Iterables.concat;
+
+/**
+ * This class provides access to Calcite's {@link PruneEmptyRules}.
+ * The instances of the rules use {@link
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder}.
+ */
+public class HiveRemoveEmptySingleRules extends PruneEmptyRules {
+
+ public static final RelOptRule PROJECT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyProject")
+ .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ .withOperandFor(Project.class, project -> true)
Review Comment:
One thing to keep in mind is that the rule may turn `ConventionXx` to
`HiveXX` since we use the `HIVE_BUILDER`. If you think that it is safe to do so
here then I am fine with it.
Issue Time Tracking
---
Worklog Id: (was: 813666)
Time Spent: 5h 10m (was: 5h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813661=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813661
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 10:57
Start Date: 30/Sep/22 10:57
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r984472010
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRemoveEmptySingleRules.java:
##
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Join;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rel.core.Union;
+import org.apache.calcite.rel.core.Values;
+import org.apache.calcite.rel.rules.PruneEmptyRules;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.hadoop.hive.ql.optimizer.calcite.Bug;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+
+import java.util.Collections;
+import java.util.List;
+
+import static com.google.common.collect.Iterables.concat;
+
+/**
+ * This class provides access to Calcite's {@link PruneEmptyRules}.
+ * The instances of the rules use {@link
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder}.
+ */
+public class HiveRemoveEmptySingleRules extends PruneEmptyRules {
+
+ public static final RelOptRule PROJECT_INSTANCE =
+ RelRule.Config.EMPTY
+ .withDescription("HivePruneEmptyProject")
+ .as(PruneEmptyRules.RemoveEmptySingleRule.Config.class)
+ .withOperandFor(Project.class, project -> true)
Review Comment:
I tried some tests locally and it works with Jdbc operators. I think it
makes sense to match general operators.
Issue Time Tracking
---
Worklog Id: (was: 813661)
Time Spent: 4h 50m (was: 4h 40m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813662
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 10:57
Start Date: 30/Sep/22 10:57
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r984472066
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java:
##
@@ -121,7 +127,87 @@ public static ASTNode convert(final RelNode relNode,
List resultSch
return c.convert();
}
+ //TOK_QUERY
+ // TOK_INSERT
+ // TOK_DESTINATION
+ //TOK_DIR
+ // TOK_TMP_FILE
+ // TOK_SELECT
+ //TOK_SELEXPR
+ // TOK_FUNCTION
+ // TOK_
+ // TOK_NULL
+ // alias0
+ //...
+ //TOK_SELEXPR
+ // TOK_FUNCTION
+ // TOK_
+ // TOK_NULL
+ // aliasn
+ // TOK_LIMIT
+ //0
+ //0
+ public static ASTNode emptyPlan(RelDataType dataType) {
+if (dataType.getFieldCount() == 0) {
+ throw new IllegalArgumentException("Schema is empty.");
+}
+
+ASTBuilder select = ASTBuilder.construct(HiveParser.TOK_SELECT,
"TOK_SELECT");
+for (int i = 0; i < dataType.getFieldCount(); ++i) {
+ RelDataTypeField fieldType = dataType.getFieldList().get(i);
+ if (fieldType.getValue().getSqlTypeName() == SqlTypeName.NULL) {
+select.add(ASTBuilder.selectExpr(
+ASTBuilder.construct(HiveParser.TOK_NULL, "TOK_NULL").node(),
+fieldType.getName()));
+ } else {
+ASTNode typeNode = createCast(fieldType);
+select.add(ASTBuilder.selectExpr(
+ASTBuilder.construct(HiveParser.TOK_FUNCTION, "TOK_FUNCTION")
+.add(typeNode)
+.add(ASTBuilder.construct(HiveParser.TOK_NULL,
"TOK_NULL").node()).node(),
+fieldType.getName()));
+ }
+}
+
+ASTNode insert = ASTBuilder.
+construct(HiveParser.TOK_INSERT, "TOK_INSERT").
+add(ASTBuilder.destNode()).
+add(select).
+add(ASTBuilder.limit(0, 0)).
+node();
+
+return ASTBuilder.
+construct(HiveParser.TOK_QUERY, "TOK_QUERY").
+add(insert).
+node();
+ }
+
+ private static ASTNode createCast(RelDataTypeField fieldType) {
+HiveToken ht = TypeConverter.hiveToken(fieldType.getType());
+ASTNode typeNode;
+if (ht == null) {
+ typeNode = ASTBuilder.construct(
+ HiveParser.Identifier,
fieldType.getType().getSqlTypeName().getName().toLowerCase()).node();
+} else {
+ ASTBuilder typeNodeBuilder = ASTBuilder.construct(ht.type, ht.text);
+ if (ht.args != null) {
+for (String castArg : ht.args) {
+ typeNodeBuilder.add(HiveParser.Identifier, castArg);
+}
+ }
+ typeNode = typeNodeBuilder.node();
+}
+return typeNode;
+ }
+
private ASTNode convert() throws CalciteSemanticException {
+if (root instanceof HiveValues) {
+ HiveValues values = (HiveValues) root;
+ if (isEmpty(values)) {
+select = values;
+return emptyPlan(values.getRowType());
+ }
Review Comment:
Exception or assertion both are fine with me; as you prefer as long as it
fails fast :)
Issue Time Tracking
---
Worklog Id: (was: 813662)
Time Spent: 5h (was: 4h 50m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813657=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813657
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 10:46
Start Date: 30/Sep/22 10:46
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r984464007
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java:
##
@@ -121,7 +127,87 @@ public static ASTNode convert(final RelNode relNode,
List resultSch
return c.convert();
}
+ //TOK_QUERY
+ // TOK_INSERT
+ // TOK_DESTINATION
+ //TOK_DIR
+ // TOK_TMP_FILE
+ // TOK_SELECT
+ //TOK_SELEXPR
+ // TOK_FUNCTION
+ // TOK_
+ // TOK_NULL
+ // alias0
+ //...
+ //TOK_SELEXPR
+ // TOK_FUNCTION
+ // TOK_
+ // TOK_NULL
+ // aliasn
+ // TOK_LIMIT
+ //0
+ //0
+ public static ASTNode emptyPlan(RelDataType dataType) {
+if (dataType.getFieldCount() == 0) {
+ throw new IllegalArgumentException("Schema is empty.");
+}
+
+ASTBuilder select = ASTBuilder.construct(HiveParser.TOK_SELECT,
"TOK_SELECT");
+for (int i = 0; i < dataType.getFieldCount(); ++i) {
+ RelDataTypeField fieldType = dataType.getFieldList().get(i);
+ if (fieldType.getValue().getSqlTypeName() == SqlTypeName.NULL) {
+select.add(ASTBuilder.selectExpr(
+ASTBuilder.construct(HiveParser.TOK_NULL, "TOK_NULL").node(),
+fieldType.getName()));
+ } else {
+ASTNode typeNode = createCast(fieldType);
+select.add(ASTBuilder.selectExpr(
+ASTBuilder.construct(HiveParser.TOK_FUNCTION, "TOK_FUNCTION")
+.add(typeNode)
+.add(ASTBuilder.construct(HiveParser.TOK_NULL,
"TOK_NULL").node()).node(),
+fieldType.getName()));
+ }
+}
+
+ASTNode insert = ASTBuilder.
+construct(HiveParser.TOK_INSERT, "TOK_INSERT").
+add(ASTBuilder.destNode()).
+add(select).
+add(ASTBuilder.limit(0, 0)).
+node();
+
+return ASTBuilder.
+construct(HiveParser.TOK_QUERY, "TOK_QUERY").
+add(insert).
+node();
+ }
+
+ private static ASTNode createCast(RelDataTypeField fieldType) {
+HiveToken ht = TypeConverter.hiveToken(fieldType.getType());
+ASTNode typeNode;
+if (ht == null) {
+ typeNode = ASTBuilder.construct(
+ HiveParser.Identifier,
fieldType.getType().getSqlTypeName().getName().toLowerCase()).node();
+} else {
+ ASTBuilder typeNodeBuilder = ASTBuilder.construct(ht.type, ht.text);
+ if (ht.args != null) {
+for (String castArg : ht.args) {
+ typeNodeBuilder.add(HiveParser.Identifier, castArg);
+}
+ }
+ typeNode = typeNodeBuilder.node();
+}
+return typeNode;
+ }
+
private ASTNode convert() throws CalciteSemanticException {
+if (root instanceof HiveValues) {
+ HiveValues values = (HiveValues) root;
+ if (isEmpty(values)) {
+select = values;
+return emptyPlan(values.getRowType());
+ }
Review Comment:
No it can't. Theoretically non-empty `HiveValues` are not created currently.
How about throwing an exception? My problem with assertions is that it does
not evaluates in production environments and I had some difficulties to repro
issues locally from q test: so the q test failed with assertion error but
something different exception was thrown from a different place in prod.
Issue Time Tracking
---
Worklog Id: (was: 813657)
Time Spent: 4h 40m (was: 4.5h)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
>
[jira] [Work logged] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?focusedWorklogId=813654=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813654
]
ASF GitHub Bot logged work on HIVE-26320:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 10:40
Start Date: 30/Sep/22 10:40
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3628:
URL: https://github.com/apache/hive/pull/3628#discussion_r984418472
##
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java:
##
@@ -481,12 +485,36 @@ protected BytesWritable convert(Binary binary) {
},
ESTRING_CONVERTER(String.class) {
@Override
-PrimitiveConverter getConverter(final PrimitiveType type, final int index,
final ConverterParent parent, TypeInfo hiveTypeInfo) {
+PrimitiveConverter getConverter(final PrimitiveType type, final int index,
final ConverterParent parent,
+TypeInfo hiveTypeInfo) {
+ // If we have type information, we should return properly typed strings.
However, there are a variety
+ // of code paths that do not provide the typeInfo in those cases we
default to Text. This idiom is also
+ // followed by for example the BigDecimal converter in which if there is
no type information,
+ // it defaults to the widest representation
+ if (hiveTypeInfo != null) {
+String typeName = hiveTypeInfo.getTypeName().toLowerCase();
+if (typeName.startsWith(serdeConstants.CHAR_TYPE_NAME)) {
+ return new BinaryConverter(type, parent, index) {
+@Override
+ protected HiveCharWritable convert(Binary binary) {
+return new HiveCharWritable(binary.getBytes(), ((CharTypeInfo)
hiveTypeInfo).getLength());
+ }
+ };
+} else if (typeName.startsWith(serdeConstants.VARCHAR_TYPE_NAME)) {
+ return new BinaryConverter(type, parent, index)
{
+@Override
+ protected HiveVarcharWritable convert(Binary binary) {
+return new HiveVarcharWritable(binary.getBytes(),
((VarcharTypeInfo) hiveTypeInfo).getLength());
+ }
+ };
+}
+ }
Review Comment:
I think the most common way to do something based on a primitive type is to
use its category. It seems robust and more efficient.
```java
if (hiveTypeInfo instanceof PrimitiveTypeInfo) {
PrimitiveTypeInfo t = (PrimitiveTypeInfo) hiveTypeInfo;
switch (t.getPrimitiveCategory()) {
case VARCHAR:
break;
case CHAR:
break;
}
}
```
I see that in this class they rely much on the name and serdeConstants so I
am also OK with the current code as it is.
##
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java:
##
@@ -212,23 +216,59 @@ public void
testGetInt64NanosAdjustedToUTCTimestampConverter() throws Exception
}
@Test
- public void testGetTextConverter() throws Exception {
+ public void testGetTextConverterForString() throws Exception {
PrimitiveType primitiveType = Types.optional(PrimitiveTypeName.BINARY)
-.as(LogicalTypeAnnotation.stringType()).named("value");
-Writable writable = getWritableFromBinaryConverter(new VarcharTypeInfo(),
primitiveType,
-Binary.fromString("this_is_a_value"));
-Text textWritable = (Text) writable;
-assertEquals("this_is_a_value", textWritable.toString());
+.as(LogicalTypeAnnotation.stringType()).named("value");
+String value = "this_is_a_value";
+Text textWritable = (Text) getWritableFromBinaryConverter(stringTypeInfo,
primitiveType,
+Binary.fromString(value));
+// we should get what we put in
+assertEquals(value, textWritable.toString());
+ }
+
+ @Test
+ public void testGetTextConverterForChar() throws Exception {
+PrimitiveType primitiveType = Types.optional(PrimitiveTypeName.BINARY)
+.as(LogicalTypeAnnotation.stringType()).named("value");
+String value = "this_is_a_value";
+
+HiveCharWritable textWritable = (HiveCharWritable)
getWritableFromBinaryConverter(
+new CharTypeInfo(value.length() + 2), primitiveType,
Binary.fromString(value));
+// check that it enforces the length (it should be padded)
+assertEquals(value + " ", textWritable.toString());
+
+textWritable = (HiveCharWritable) getWritableFromBinaryConverter(
+new CharTypeInfo(6), primitiveType, Binary.fromString(value));
+// check that it enforces the length (it should truncate)
+assertEquals(value.substring(0, 6), textWritable.toString());
Review Comment:
nit. The test could be easily split in two and a proper name could avoid the
need for extra comments
* testGetTextConverterForCharPadsValueWithSpacesTillLen
*
[jira] [Work logged] (HIVE-26576) Alter table calls on Iceberg tables can inadvertently change metadata_location
[
https://issues.apache.org/jira/browse/HIVE-26576?focusedWorklogId=813649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813649
]
ASF GitHub Bot logged work on HIVE-26576:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 10:20
Start Date: 30/Sep/22 10:20
Worklog Time Spent: 10m
Work Description: ayushtkn commented on code in PR #3612:
URL: https://github.com/apache/hive/pull/3612#discussion_r984439963
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##
@@ -386,6 +405,13 @@ private void
assertNotCrossTableMetadataLocationChange(Map tblPa
icebergTable.name(), newMetadataLocation)
);
}
+ if (!currentMetadata.metadataFileLocation().equals(newMetadataLocation)
&&
+
!context.getProperties().containsKey(MANUAL_ICEBERG_METADATA_LOCATION_CHANGE)) {
+// If we got here then this is an alter table operation where the
table to be changed had an Iceberg commit
+// meanwhile. The base metadata locations differ, while we know that
this wasn't an intentional, manual
+// metadata_location set by a user. To protect the interim commit we
need to refresh the metadata file location.
+tblParams.put(BaseMetastoreTableOperations.METADATA_LOCATION_PROP,
currentMetadata.metadataFileLocation());
Review Comment:
Good to have a log line here, as a follow up as we discussed we can have
some logic to figure out whether it is only a Stats upgrade & if so, make this
Alter a NO-OP in future
Issue Time Tracking
---
Worklog Id: (was: 813649)
Time Spent: 40m (was: 0.5h)
> Alter table calls on Iceberg tables can inadvertently change metadata_location
> --
>
> Key: HIVE-26576
> URL: https://issues.apache.org/jira/browse/HIVE-26576
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Concurrent alter_table calls can interfere and can cause the
> metadata_location property of an Iceberg table to be messed up.
> Basically there's no table level locking for Iceberg tables in Hive during
> the usual operations, and thus some extra performance related features are
> available, like concurrent inserts, as opposed to native Hive tables. This
> was done under the assumption that the optimistic locking pattern that is
> used in HiveTableOperations protects changing the metadata_location by the
> use of an HMS table lock there only.
> This is fine until some other alter_table calls get into the system such as
> one from StatTask or DDLTask. Such tasks perform their work as:
> * get the current table
> * do the alteration
> * send the changes via alter_table call to HMS
> In between the retrieval of the table and the alter_table call a legit commit
> from HiveTableOperations might bump the metadata_location, but this will get
> reverted as these tasks consider an outdated metadata_location (and the alter
> table call will overwrite all table props including this one too..)
> This is a design issue, and to solve this while preserving the concurrency
> features I propose to make use of HiveIcebergMetaHook where all such
> alter_table calls are intercepted, and the same locking mechanism could be
> used there as the one found in HiveTableOperations. The proposed flow on HMS
> client side would be:
> * hook: preAlterTable
> ** request table level lock
> ** refresh the Iceberg table from catalog (HMS) to see if new updates have
> arrived
> ** compare the current metadata with the one thought to be the base of this
> request, if metadata_location is outdated overwrite it with the fresh,
> current one in this request
> * do the alter_table call to HMS with the relevant changes (updated stats or
> other properties)
> * hook: post/rollbackAlterTable
> ** release table level lock
> This can work as the metadata_location should never be changed by anything
> other than HiveTableOperations, which is the only thing not using this hook
> (if it did we'd be in an endless loop). There's actually one exception which
> is if a user wants to change the metadata_location by hand. I can make an
> exception to that signalling this fact from an environmentContext instance
> when the corresponding AlterTableSetPropertiesDesc is constructed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?focusedWorklogId=813638=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813638
]
ASF GitHub Bot logged work on HIVE-26320:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 09:35
Start Date: 30/Sep/22 09:35
Worklog Time Spent: 10m
Work Description: asolimando commented on code in PR #3628:
URL: https://github.com/apache/hive/pull/3628#discussion_r984400218
##
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java:
##
@@ -212,23 +216,59 @@ public void
testGetInt64NanosAdjustedToUTCTimestampConverter() throws Exception
}
@Test
- public void testGetTextConverter() throws Exception {
+ public void testGetTextConverterForString() throws Exception {
PrimitiveType primitiveType = Types.optional(PrimitiveTypeName.BINARY)
-.as(LogicalTypeAnnotation.stringType()).named("value");
-Writable writable = getWritableFromBinaryConverter(new VarcharTypeInfo(),
primitiveType,
-Binary.fromString("this_is_a_value"));
-Text textWritable = (Text) writable;
-assertEquals("this_is_a_value", textWritable.toString());
+.as(LogicalTypeAnnotation.stringType()).named("value");
+String value = "this_is_a_value";
+Text textWritable = (Text) getWritableFromBinaryConverter(stringTypeInfo,
primitiveType,
+Binary.fromString(value));
+// we should get what we put in
+assertEquals(value, textWritable.toString());
+ }
+
+ @Test
+ public void testGetTextConverterForChar() throws Exception {
Review Comment:
Nit: same here, `throws Exception` is redundant
```suggestion
public void testGetTextConverterForChar() {
```
##
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java:
##
@@ -212,23 +216,59 @@ public void
testGetInt64NanosAdjustedToUTCTimestampConverter() throws Exception
}
@Test
- public void testGetTextConverter() throws Exception {
+ public void testGetTextConverterForString() throws Exception {
PrimitiveType primitiveType = Types.optional(PrimitiveTypeName.BINARY)
-.as(LogicalTypeAnnotation.stringType()).named("value");
-Writable writable = getWritableFromBinaryConverter(new VarcharTypeInfo(),
primitiveType,
-Binary.fromString("this_is_a_value"));
-Text textWritable = (Text) writable;
-assertEquals("this_is_a_value", textWritable.toString());
+.as(LogicalTypeAnnotation.stringType()).named("value");
+String value = "this_is_a_value";
+Text textWritable = (Text) getWritableFromBinaryConverter(stringTypeInfo,
primitiveType,
+Binary.fromString(value));
+// we should get what we put in
+assertEquals(value, textWritable.toString());
+ }
+
+ @Test
+ public void testGetTextConverterForChar() throws Exception {
+PrimitiveType primitiveType = Types.optional(PrimitiveTypeName.BINARY)
+.as(LogicalTypeAnnotation.stringType()).named("value");
+String value = "this_is_a_value";
+
+HiveCharWritable textWritable = (HiveCharWritable)
getWritableFromBinaryConverter(
+new CharTypeInfo(value.length() + 2), primitiveType,
Binary.fromString(value));
+// check that it enforces the length (it should be padded)
+assertEquals(value + " ", textWritable.toString());
+
+textWritable = (HiveCharWritable) getWritableFromBinaryConverter(
+new CharTypeInfo(6), primitiveType, Binary.fromString(value));
+// check that it enforces the length (it should truncate)
+assertEquals(value.substring(0, 6), textWritable.toString());
+ }
+
+ @Test
+ public void testGetTextConverterForVarchar() throws Exception {
Review Comment:
```suggestion
public void testGetTextConverterForVarchar() {
```
##
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java:
##
@@ -212,23 +216,59 @@ public void
testGetInt64NanosAdjustedToUTCTimestampConverter() throws Exception
}
@Test
- public void testGetTextConverter() throws Exception {
+ public void testGetTextConverterForString() throws Exception {
Review Comment:
Nit: the "throws Exception" bit is redundant and can be safely removed
```suggestion
public void testGetTextConverterForString() {
```
##
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java:
##
@@ -212,23 +216,59 @@ public void
testGetInt64NanosAdjustedToUTCTimestampConverter() throws Exception
}
@Test
- public void testGetTextConverter() throws Exception {
+ public void testGetTextConverterForString() throws Exception {
PrimitiveType primitiveType = Types.optional(PrimitiveTypeName.BINARY)
-.as(LogicalTypeAnnotation.stringType()).named("value");
-Writable writable = getWritableFromBinaryConverter(new
[jira] [Work logged] (HIVE-26576) Alter table calls on Iceberg tables can inadvertently change metadata_location
[ https://issues.apache.org/jira/browse/HIVE-26576?focusedWorklogId=813630=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813630 ] ASF GitHub Bot logged work on HIVE-26576: - Author: ASF GitHub Bot Created on: 30/Sep/22 09:21 Start Date: 30/Sep/22 09:21 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3612: URL: https://github.com/apache/hive/pull/3612#issuecomment-1263327725 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3612) [](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=BUG) [5 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3612=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3612=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3612=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3612=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3612=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3612=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 813630) Time Spent: 0.5h (was: 20m) > Alter table calls on Iceberg tables can inadvertently change metadata_location > -- > > Key: HIVE-26576 > URL: https://issues.apache.org/jira/browse/HIVE-26576 > Project: Hive > Issue Type: Bug >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Concurrent alter_table calls can interfere and can cause the > metadata_location property of an Iceberg table to be messed up. > Basically there's no table level locking for Iceberg tables in Hive during > the usual operations, and thus some extra performance related features are > available, like concurrent inserts, as opposed to native Hive tables. This > was done under the assumption that the optimistic locking pattern that is > used in HiveTableOperations protects changing the metadata_location by the > use of an HMS table lock there only. > This is fine until some other alter_table calls get into the system such as > one from StatTask or DDLTask. Such tasks perform their work as: > * get the current table > * do the alteration > * send the changes via alter_table call to HMS > In between the retrieval of the table and the alter_table call a legit commit > from HiveTableOperations might bump the metadata_location, but this will get >
[jira] [Resolved] (HIVE-26529) Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat
[ https://issues.apache.org/jira/browse/HIVE-26529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-26529. - Resolution: Fixed > Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat > --- > > Key: HIVE-26529 > URL: https://issues.apache.org/jira/browse/HIVE-26529 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: perfomance, pull-request-available > Attachments: iceberg_table_with_HiveDecimal.png, > regular_tables_with_decimal64.png > > Time Spent: 5.5h > Remaining Estimate: 0h > > For supporting vectored reads in parquet, DECIMAL_64 support in ORC has been > disabled in HiveIcebergInputFormat. This causes regressions in queries. > [https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java#L182] > It will be good to restore DECIMAL_64 support in iceberg input format. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26529) Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
[ https://issues.apache.org/jira/browse/HIVE-26529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-26529: Summary: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat. (was: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat) > Iceberg: Fix VectorizedSupport support for DECIMAL_64 in > HiveIcebergInputFormat. > > > Key: HIVE-26529 > URL: https://issues.apache.org/jira/browse/HIVE-26529 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: perfomance, pull-request-available > Attachments: iceberg_table_with_HiveDecimal.png, > regular_tables_with_decimal64.png > > Time Spent: 5.5h > Remaining Estimate: 0h > > For supporting vectored reads in parquet, DECIMAL_64 support in ORC has been > disabled in HiveIcebergInputFormat. This causes regressions in queries. > [https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java#L182] > It will be good to restore DECIMAL_64 support in iceberg input format. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26529) Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
[ https://issues.apache.org/jira/browse/HIVE-26529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-26529: Fix Version/s: 4.0.0-alpha-2 > Iceberg: Fix VectorizedSupport support for DECIMAL_64 in > HiveIcebergInputFormat. > > > Key: HIVE-26529 > URL: https://issues.apache.org/jira/browse/HIVE-26529 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: perfomance, pull-request-available > Fix For: 4.0.0-alpha-2 > > Attachments: iceberg_table_with_HiveDecimal.png, > regular_tables_with_decimal64.png > > Time Spent: 5.5h > Remaining Estimate: 0h > > For supporting vectored reads in parquet, DECIMAL_64 support in ORC has been > disabled in HiveIcebergInputFormat. This causes regressions in queries. > [https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java#L182] > It will be good to restore DECIMAL_64 support in iceberg input format. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26529) Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat
[ https://issues.apache.org/jira/browse/HIVE-26529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611456#comment-17611456 ] Ayush Saxena commented on HIVE-26529: - Committed to master. Thanx [~rajesh.balamohan] and [~szita]!!! > Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat > --- > > Key: HIVE-26529 > URL: https://issues.apache.org/jira/browse/HIVE-26529 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: perfomance, pull-request-available > Attachments: iceberg_table_with_HiveDecimal.png, > regular_tables_with_decimal64.png > > Time Spent: 5.5h > Remaining Estimate: 0h > > For supporting vectored reads in parquet, DECIMAL_64 support in ORC has been > disabled in HiveIcebergInputFormat. This causes regressions in queries. > [https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java#L182] > It will be good to restore DECIMAL_64 support in iceberg input format. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26529) Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat
[ https://issues.apache.org/jira/browse/HIVE-26529?focusedWorklogId=813627=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813627 ] ASF GitHub Bot logged work on HIVE-26529: - Author: ASF GitHub Bot Created on: 30/Sep/22 09:17 Start Date: 30/Sep/22 09:17 Worklog Time Spent: 10m Work Description: ayushtkn merged PR #3607: URL: https://github.com/apache/hive/pull/3607 Issue Time Tracking --- Worklog Id: (was: 813627) Time Spent: 5.5h (was: 5h 20m) > Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat > --- > > Key: HIVE-26529 > URL: https://issues.apache.org/jira/browse/HIVE-26529 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: perfomance, pull-request-available > Attachments: iceberg_table_with_HiveDecimal.png, > regular_tables_with_decimal64.png > > Time Spent: 5.5h > Remaining Estimate: 0h > > For supporting vectored reads in parquet, DECIMAL_64 support in ORC has been > disabled in HiveIcebergInputFormat. This causes regressions in queries. > [https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java#L182] > It will be good to restore DECIMAL_64 support in iceberg input format. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26524) Use Calcite to remove sections of a query plan known never produces rows
[
https://issues.apache.org/jira/browse/HIVE-26524?focusedWorklogId=813623=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813623
]
ASF GitHub Bot logged work on HIVE-26524:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 09:11
Start Date: 30/Sep/22 09:11
Worklog Time Spent: 10m
Work Description: asolimando commented on code in PR #3588:
URL: https://github.com/apache/hive/pull/3588#discussion_r984386036
##
ql/src/test/results/clientpositive/llap/float_equality.q.out:
##
@@ -9,9 +9,7 @@ POSTHOOK: Input: _dummy_database@_dummy_table
1
PREHOOK: query: select 1 where -0.0<0.0
PREHOOK: type: QUERY
-PREHOOK: Input: _dummy_database@_dummy_table
Review Comment:
In my understanding the scan is involved in an empty sub-plan that now is
not executed, so the scan does not take place.
Issue Time Tracking
---
Worklog Id: (was: 813623)
Time Spent: 4.5h (was: 4h 20m)
> Use Calcite to remove sections of a query plan known never produces rows
>
>
> Key: HIVE-26524
> URL: https://issues.apache.org/jira/browse/HIVE-26524
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Calcite has a set of rules to remove sections of a query plan known never
> produces any rows. In some cases the whole plan can be removed. Such plans
> are represented with a single {{Values}} operators with no tuples. ex.:
> {code:java}
> select y + 1 from (select a1 y, b1 z from t1 where b1 > 10) q WHERE 1=0
> {code}
> {code:java}
> HiveValues(tuples=[[]])
> {code}
> Other cases when plan has outer join or set operators some branches can be
> replaced with empty values moving forward in some cases the join/set operator
> can be removed
> {code:java}
> select a2, b2 from t2 where 1=0
> union
> select a1, b1 from t1
> {code}
> {code:java}
> HiveAggregate(group=[{0, 1}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26578) Enable Iceberg storage format for materialized views
[
https://issues.apache.org/jira/browse/HIVE-26578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa reassigned HIVE-26578:
-
> Enable Iceberg storage format for materialized views
>
>
> Key: HIVE-26578
> URL: https://issues.apache.org/jira/browse/HIVE-26578
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
>
> {code}
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26529) Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat
[
https://issues.apache.org/jira/browse/HIVE-26529?focusedWorklogId=813591=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813591
]
ASF GitHub Bot logged work on HIVE-26529:
-
Author: ASF GitHub Bot
Created on: 30/Sep/22 06:51
Start Date: 30/Sep/22 06:51
Worklog Time Spent: 10m
Work Description: ayushtkn commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r984264418
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java:
##
@@ -177,9 +178,19 @@ public boolean shouldSkipCombine(Path path, Configuration
conf) {
@Override
public VectorizedSupport.Support[] getSupportedFeatures() {
-// disabling VectorizedSupport.Support.DECIMAL_64 as Parquet doesn't
support it, and we have no way of telling
-// beforehand what kind of file format we're going to hit later
-return new VectorizedSupport.Support[]{ };
+throw new UnsupportedOperationException("This overload of
getSupportedFeatures should never be called");
+ }
+
+ @Override
+ public VectorizedSupport.Support[] getSupportedFeatures(HiveConf hiveConf,
Properties properties) {
+// disabling VectorizedSupport.Support.DECIMAL_64 for Parquet as it
doesn't support it
+boolean isORCOnly =
Boolean.parseBoolean(properties.getProperty(HiveIcebergMetaHook.ORC_FILES_ONLY));
Review Comment:
Done
Issue Time Tracking
---
Worklog Id: (was: 813591)
Time Spent: 5h 20m (was: 5h 10m)
> Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat
> ---
>
> Key: HIVE-26529
> URL: https://issues.apache.org/jira/browse/HIVE-26529
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Ayush Saxena
>Priority: Major
> Labels: perfomance, pull-request-available
> Attachments: iceberg_table_with_HiveDecimal.png,
> regular_tables_with_decimal64.png
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> For supporting vectored reads in parquet, DECIMAL_64 support in ORC has been
> disabled in HiveIcebergInputFormat. This causes regressions in queries.
> [https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java#L182]
> It will be good to restore DECIMAL_64 support in iceberg input format.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26529) Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat
[ https://issues.apache.org/jira/browse/HIVE-26529?focusedWorklogId=813590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-813590 ] ASF GitHub Bot logged work on HIVE-26529: - Author: ASF GitHub Bot Created on: 30/Sep/22 06:46 Start Date: 30/Sep/22 06:46 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3607: URL: https://github.com/apache/hive/pull/3607#issuecomment-1263174289 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3607) [](https://sonarcloud.io/project/issues?id=apache_hive=3607=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3607=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3607=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3607=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3607=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3607=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3607=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3607=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3607=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3607=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3607=false=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3607=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3607=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3607=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 813590) Time Spent: 5h 10m (was: 5h) > Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat > --- > > Key: HIVE-26529 > URL: https://issues.apache.org/jira/browse/HIVE-26529 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: perfomance, pull-request-available > Attachments: iceberg_table_with_HiveDecimal.png, > regular_tables_with_decimal64.png > > Time Spent: 5h 10m > Remaining Estimate: 0h > > For supporting vectored reads in parquet, DECIMAL_64 support in ORC has been > disabled in HiveIcebergInputFormat. This causes regressions in queries. > [https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java#L182] > It will be good to restore DECIMAL_64 support in iceberg input format. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
