[jira] [Work logged] (HIVE-26794) Explore changing TxnHandler#connPoolMutex to NoPoolConnectionPool
[ https://issues.apache.org/jira/browse/HIVE-26794?focusedWorklogId=830823=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830823 ] ASF GitHub Bot logged work on HIVE-26794: - Author: ASF GitHub Bot Created on: 03/Dec/22 07:44 Start Date: 03/Dec/22 07:44 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3817: URL: https://github.com/apache/hive/pull/3817#issuecomment-1336106336 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3817) [](https://sonarcloud.io/project/issues?id=apache_hive=3817=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3817=false=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive=3817=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3817=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3817=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3817=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3817=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3817=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3817=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3817=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3817=false=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3817=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3817=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3817=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 830823) Time Spent: 1.5h (was: 1h 20m) > Explore changing TxnHandler#connPoolMutex to NoPoolConnectionPool > - > > Key: HIVE-26794 > URL: https://issues.apache.org/jira/browse/HIVE-26794 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Instead of creating a fixed size connection pool for TxnHandler#MutexAPI, the > pool can be assigned to NoPoolConnectionPool due to: > * TxnHandler#MutexAPI is primarily designed to provide coarse-grained mutex > support to maintenance tasks running inside the Metastore, these tasks are > not user faced; > * A fixed size connection pool as same as the pool used in ObjectStore is a > waste for other non leaders in the warehouse; > The NoPoolConnectionPool provides connection on demand, and > TxnHandler#MutexAPI only uses getConnection method to fetch a connection from > the pool, so it's doable to change the pool to NoPoolConnectionPool, this > would make the HMS more scaleable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26801) Query based compaction fails on tables having columns with keywords(i.e. row in this case)
[
https://issues.apache.org/jira/browse/HIVE-26801?focusedWorklogId=830821=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830821
]
ASF GitHub Bot logged work on HIVE-26801:
-
Author: ASF GitHub Bot
Created on: 03/Dec/22 07:31
Start Date: 03/Dec/22 07:31
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3828:
URL: https://github.com/apache/hive/pull/3828#issuecomment-1336104551
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3828)
[](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3828=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3828=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3828=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3828=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3828=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 830821)
Time Spent: 0.5h (was: 20m)
> Query based compaction fails on tables having columns with keywords(i.e. row
> in this case)
> --
>
> Key: HIVE-26801
> URL: https://issues.apache.org/jira/browse/HIVE-26801
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Gopinath
>Assignee: Gopinath
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Query-based compaction fails on tables having columns with keywords(i.e. row
> in this case) for columns. The compaction fails while running insert into
> statement as it does not quote the columns correctly.
> Below are the steps to reproduce the issue.
> {noformat}
> CREATE TABLE aggregated_data(`sessionid` string,`row` int,`timeofoccurrence`
> bigint);
> insert into table aggregated_data values ("abcd",300,211);
> insert into table aggregated_data values ("abcd",300,211);
> alter table aggregated_data compact 'MAJOR' and wait;{noformat}
> Error -
> {noformat}
> 2022-11-30 13:04:33,309 INFO
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor: [repro894918]:
> Running major compaction via query: INSERT into table
> default_tmp_compactor_aggregated_data_1669813472898 select
> validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
> ROW__ID.writeId,
[jira] [Work logged] (HIVE-26692) Check for the expected thrift version before compiling
[ https://issues.apache.org/jira/browse/HIVE-26692?focusedWorklogId=830816=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830816 ] ASF GitHub Bot logged work on HIVE-26692: - Author: ASF GitHub Bot Created on: 03/Dec/22 06:41 Start Date: 03/Dec/22 06:41 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3820: URL: https://github.com/apache/hive/pull/3820#issuecomment-1336096678 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3820) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3820=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3820=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3820=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3820=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3820=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 830816) Time Spent: 2h 10m (was: 2h) > Check for the expected thrift version before compiling > -- > > Key: HIVE-26692 > URL: https://issues.apache.org/jira/browse/HIVE-26692 > Project: Hive > Issue Type: Task > Components: Thrift API >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > At the moment we don't check for the thrift version before launching thrift, > the error messages are often cryptic upon mismatches. > An explicit check with a clear error message would be nice, like what parquet > does: > [https://github.com/apache/parquet-mr/blob/master/parquet-thrift/pom.xml#L247-L268] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26685) Improve Path name escaping / unescaping performance
[ https://issues.apache.org/jira/browse/HIVE-26685?focusedWorklogId=830799=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830799 ] ASF GitHub Bot logged work on HIVE-26685: - Author: ASF GitHub Bot Created on: 03/Dec/22 02:24 Start Date: 03/Dec/22 02:24 Worklog Time Spent: 10m Work Description: weiatwork commented on PR #3721: URL: https://github.com/apache/hive/pull/3721#issuecomment-1336016752 +1 Issue Time Tracking --- Worklog Id: (was: 830799) Time Spent: 0.5h (was: 20m) > Improve Path name escaping / unescaping performance > --- > > Key: HIVE-26685 > URL: https://issues.apache.org/jira/browse/HIVE-26685 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: All Versions >Reporter: James Petty >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-26685.1.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > When escaping / unescaping partition path part names, the existing logic > incurs significant avoidable overhead by copying each character sequentially > into a new StringBuilder even when no escaping/unescaping is necessary as > well as using String.format to escape characters inside of the inner loop. > > The included patch to improve the performance of these operations refactors > two static method implementations, but requires no external API surface or > user-visible behavior changes. This change is applicable and portable to a > wide range of Hive versions from branch-0.6 onward when the initial method > implementations were added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26685) Improve Path name escaping / unescaping performance
[ https://issues.apache.org/jira/browse/HIVE-26685?focusedWorklogId=830798=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830798 ] ASF GitHub Bot logged work on HIVE-26685: - Author: ASF GitHub Bot Created on: 03/Dec/22 02:24 Start Date: 03/Dec/22 02:24 Worklog Time Spent: 10m Work Description: weiatwork commented on PR #3721: URL: https://github.com/apache/hive/pull/3721#issuecomment-1336016723 That's a nice perf improvement! Issue Time Tracking --- Worklog Id: (was: 830798) Time Spent: 20m (was: 10m) > Improve Path name escaping / unescaping performance > --- > > Key: HIVE-26685 > URL: https://issues.apache.org/jira/browse/HIVE-26685 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: All Versions >Reporter: James Petty >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-26685.1.patch > > Time Spent: 20m > Remaining Estimate: 0h > > When escaping / unescaping partition path part names, the existing logic > incurs significant avoidable overhead by copying each character sequentially > into a new StringBuilder even when no escaping/unescaping is necessary as > well as using String.format to escape characters inside of the inner loop. > > The included patch to improve the performance of these operations refactors > two static method implementations, but requires no external API surface or > user-visible behavior changes. This change is applicable and portable to a > wide range of Hive versions from branch-0.6 onward when the initial method > implementations were added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26683) Sum over window produces 0 when row contains null
[
https://issues.apache.org/jira/browse/HIVE-26683?focusedWorklogId=830758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830758
]
ASF GitHub Bot logged work on HIVE-26683:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 23:22
Start Date: 02/Dec/22 23:22
Worklog Time Spent: 10m
Work Description: scarlin-cloudera commented on code in PR #3800:
URL: https://github.com/apache/hive/pull/3800#discussion_r1038620856
##
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java:
##
@@ -337,29 +345,35 @@ public GenericUDAFEvaluator
getWindowingEvaluator(WindowFrameDef wFrameDef) {
return null;
}
- return new
GenericUDAFStreamingEvaluator.SumAvgEnhancer(
+ return new
GenericUDAFStreamingEvaluator.SumAvgEnhancer(
Review Comment:
This matches very similar logic found in GenericUDAFAvg which also uses an
Object[]
The purpose is that we need to keep track of the sum while also keeping
track whether the current window member is a null. If all members in the
window frame are null, we should treat the sum as null instead of 0, which is
how the "preceding" rows behave.
Issue Time Tracking
---
Worklog Id: (was: 830758)
Time Spent: 1h 40m (was: 1.5h)
> Sum over window produces 0 when row contains null
> -
>
> Key: HIVE-26683
> URL: https://issues.apache.org/jira/browse/HIVE-26683
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Steve Carlin
>Assignee: Steve Carlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Ran the following sql:
>
> {code:java}
> create table sum_window_test_small (id int, tinyint_col tinyint);
> insert into sum_window_test_small values (5,5), (10, NULL), (11,1);
> select id,
> tinyint_col,
> sum(tinyint_col) over (order by id nulls last rows between 1 following and 1
> following)
> from sum_window_test_small order by id;
> select id,
> tinyint_col,
> sum(tinyint_col) over (order by id nulls last rows between current row and 1
> following)
> from sum_window_test_small order by id;
> {code}
> The result is
> {code:java}
> +-+--+---+
> | id | tinyint_col | sum_window_0 |
> +-+--+---+
> | 5 | 5 | 0 |
> | 10 | NULL | 1 |
> | 11 | 1 | NULL |
> +-+--+---+{code}
> The first row should have the sum as NULL
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26683) Sum over window produces 0 when row contains null
[
https://issues.apache.org/jira/browse/HIVE-26683?focusedWorklogId=830757=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830757
]
ASF GitHub Bot logged work on HIVE-26683:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 23:20
Start Date: 02/Dec/22 23:20
Worklog Time Spent: 10m
Work Description: ramesh0201 commented on PR #3800:
URL: https://github.com/apache/hive/pull/3800#issuecomment-1335953950
Looks good to me except the one comment
Issue Time Tracking
---
Worklog Id: (was: 830757)
Time Spent: 1.5h (was: 1h 20m)
> Sum over window produces 0 when row contains null
> -
>
> Key: HIVE-26683
> URL: https://issues.apache.org/jira/browse/HIVE-26683
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Steve Carlin
>Assignee: Steve Carlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Ran the following sql:
>
> {code:java}
> create table sum_window_test_small (id int, tinyint_col tinyint);
> insert into sum_window_test_small values (5,5), (10, NULL), (11,1);
> select id,
> tinyint_col,
> sum(tinyint_col) over (order by id nulls last rows between 1 following and 1
> following)
> from sum_window_test_small order by id;
> select id,
> tinyint_col,
> sum(tinyint_col) over (order by id nulls last rows between current row and 1
> following)
> from sum_window_test_small order by id;
> {code}
> The result is
> {code:java}
> +-+--+---+
> | id | tinyint_col | sum_window_0 |
> +-+--+---+
> | 5 | 5 | 0 |
> | 10 | NULL | 1 |
> | 11 | 1 | NULL |
> +-+--+---+{code}
> The first row should have the sum as NULL
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26683) Sum over window produces 0 when row contains null
[
https://issues.apache.org/jira/browse/HIVE-26683?focusedWorklogId=830756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830756
]
ASF GitHub Bot logged work on HIVE-26683:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 23:20
Start Date: 02/Dec/22 23:20
Worklog Time Spent: 10m
Work Description: ramesh0201 commented on code in PR #3800:
URL: https://github.com/apache/hive/pull/3800#discussion_r1038620035
##
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java:
##
@@ -337,29 +345,35 @@ public GenericUDAFEvaluator
getWindowingEvaluator(WindowFrameDef wFrameDef) {
return null;
}
- return new
GenericUDAFStreamingEvaluator.SumAvgEnhancer(
+ return new
GenericUDAFStreamingEvaluator.SumAvgEnhancer(
Review Comment:
Hi @scarlin-cloudera Why are we changing this to Object[] instead of
HiveDecimal and following places?
Issue Time Tracking
---
Worklog Id: (was: 830756)
Time Spent: 1h 20m (was: 1h 10m)
> Sum over window produces 0 when row contains null
> -
>
> Key: HIVE-26683
> URL: https://issues.apache.org/jira/browse/HIVE-26683
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Steve Carlin
>Assignee: Steve Carlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Ran the following sql:
>
> {code:java}
> create table sum_window_test_small (id int, tinyint_col tinyint);
> insert into sum_window_test_small values (5,5), (10, NULL), (11,1);
> select id,
> tinyint_col,
> sum(tinyint_col) over (order by id nulls last rows between 1 following and 1
> following)
> from sum_window_test_small order by id;
> select id,
> tinyint_col,
> sum(tinyint_col) over (order by id nulls last rows between current row and 1
> following)
> from sum_window_test_small order by id;
> {code}
> The result is
> {code:java}
> +-+--+---+
> | id | tinyint_col | sum_window_0 |
> +-+--+---+
> | 5 | 5 | 0 |
> | 10 | NULL | 1 |
> | 11 | 1 | NULL |
> +-+--+---+{code}
> The first row should have the sum as NULL
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26789) Add UserName in CallerContext for queries
[ https://issues.apache.org/jira/browse/HIVE-26789?focusedWorklogId=830752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830752 ] ASF GitHub Bot logged work on HIVE-26789: - Author: ASF GitHub Bot Created on: 02/Dec/22 23:08 Start Date: 02/Dec/22 23:08 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3813: URL: https://github.com/apache/hive/pull/3813#issuecomment-1335947329 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3813) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3813=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3813=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3813=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3813=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3813=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 830752) Time Spent: 2h 50m (was: 2h 40m) > Add UserName in CallerContext for queries > - > > Key: HIVE-26789 > URL: https://issues.apache.org/jira/browse/HIVE-26789 > Project: Hive > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > HDFS Audit logs if impersonation is false, tracks only the Hive user in the > audit log, Can pass the actual user as part of the CallerContext, so that can > be logged as well for better tracking -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26806) Precommit tests in CI are timing out after HIVE-26796
[
https://issues.apache.org/jira/browse/HIVE-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642702#comment-17642702
]
Ayush Saxena commented on HIVE-26806:
-
I changed the build timeout in 1511, without the commit, just while replaying
added the timeout. Which you aborted. :P
> Precommit tests in CI are timing out after HIVE-26796
> -
>
> Key: HIVE-26806
> URL: https://issues.apache.org/jira/browse/HIVE-26806
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> http://ci.hive.apache.org/job/hive-precommit/job/master/1506/
> {noformat}
> ancelling nested steps due to timeout
> 15:22:08 Sending interrupt signal to process
> 15:22:08 Killing processes
> 15:22:09 kill finished with exit code 0
> 15:22:19 Terminated
> 15:22:19 script returned exit code 143
> [Pipeline] }
> [Pipeline] // withEnv
> [Pipeline] }
> 15:22:19 Deleting 1 temporary files
> [Pipeline] // configFileProvider
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (PostProcess)
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] junit
> 15:22:25 Recording test results
> 15:22:32 [Checks API] No suitable checks publisher found.
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] }
> [Pipeline] // container
> [Pipeline] }
> [Pipeline] // node
> [Pipeline] }
> [Pipeline] // timeout
> [Pipeline] }
> [Pipeline] // podTemplate
> [Pipeline] }
> 15:22:32 Failed in branch split-01
> [Pipeline] // parallel
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (Archive)
> [Pipeline] podTemplate
> [Pipeline] {
> [Pipeline] timeout
> 15:22:33 Timeout set to expire in 6 hr 0 min
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26806) Precommit tests in CI are timing out after HIVE-26796
[
https://issues.apache.org/jira/browse/HIVE-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642695#comment-17642695
]
Stamatis Zampetakis commented on HIVE-26806:
The last run http://ci.hive.apache.org/job/hive-precommit/job/master/1510/ also
failed with timeout exceeded. I will double the timeout temporarily and commit
again.
> Precommit tests in CI are timing out after HIVE-26796
> -
>
> Key: HIVE-26806
> URL: https://issues.apache.org/jira/browse/HIVE-26806
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> http://ci.hive.apache.org/job/hive-precommit/job/master/1506/
> {noformat}
> ancelling nested steps due to timeout
> 15:22:08 Sending interrupt signal to process
> 15:22:08 Killing processes
> 15:22:09 kill finished with exit code 0
> 15:22:19 Terminated
> 15:22:19 script returned exit code 143
> [Pipeline] }
> [Pipeline] // withEnv
> [Pipeline] }
> 15:22:19 Deleting 1 temporary files
> [Pipeline] // configFileProvider
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (PostProcess)
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] junit
> 15:22:25 Recording test results
> 15:22:32 [Checks API] No suitable checks publisher found.
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] }
> [Pipeline] // container
> [Pipeline] }
> [Pipeline] // node
> [Pipeline] }
> [Pipeline] // timeout
> [Pipeline] }
> [Pipeline] // podTemplate
> [Pipeline] }
> 15:22:32 Failed in branch split-01
> [Pipeline] // parallel
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (Archive)
> [Pipeline] podTemplate
> [Pipeline] {
> [Pipeline] timeout
> 15:22:33 Timeout set to expire in 6 hr 0 min
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26692) Check for the expected thrift version before compiling
[ https://issues.apache.org/jira/browse/HIVE-26692?focusedWorklogId=830735=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830735 ] ASF GitHub Bot logged work on HIVE-26692: - Author: ASF GitHub Bot Created on: 02/Dec/22 20:30 Start Date: 02/Dec/22 20:30 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3820: URL: https://github.com/apache/hive/pull/3820#issuecomment-1335813090 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3820) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3820=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3820=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3820=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3820=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3820=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 830735) Time Spent: 2h (was: 1h 50m) > Check for the expected thrift version before compiling > -- > > Key: HIVE-26692 > URL: https://issues.apache.org/jira/browse/HIVE-26692 > Project: Hive > Issue Type: Task > Components: Thrift API >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > At the moment we don't check for the thrift version before launching thrift, > the error messages are often cryptic upon mismatches. > An explicit check with a clear error message would be nice, like what parquet > does: > [https://github.com/apache/parquet-mr/blob/master/parquet-thrift/pom.xml#L247-L268] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26762) Remove operand pruning in HiveFilterSetOpTransposeRule
[
https://issues.apache.org/jira/browse/HIVE-26762?focusedWorklogId=830718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830718
]
ASF GitHub Bot logged work on HIVE-26762:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 19:30
Start Date: 02/Dec/22 19:30
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3825:
URL: https://github.com/apache/hive/pull/3825#issuecomment-1335719640
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3825)
[](https://sonarcloud.io/project/issues?id=apache_hive=3825=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3825=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3825=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3825=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3825=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3825=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3825=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3825=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3825=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3825=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3825=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3825=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3825=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3825=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 830718)
Time Spent: 0.5h (was: 20m)
> Remove operand pruning in HiveFilterSetOpTransposeRule
> --
>
> Key: HIVE-26762
> URL: https://issues.apache.org/jira/browse/HIVE-26762
> Project: Hive
> Issue Type: Task
> Components: CBO, Query Planning
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> HiveFilterSetOpTransposeRule, when applied to UNION ALL operands, checks if
> the newly pushed filter simplifies to FALSE (due to the predicates holding on
> the input).
> If this is true and there is more than one UNION ALL operand, it gets pruned.
> After HIVE-26524 ("Use Calcite to remove sections of a query plan known never
> produces rows"), this is possibly redundant and we could drop this feature
> and let the other rules take care of the pruning.
> In such a case, it might be even possible to drop the Hive specific rule and
> relies on the Calcite one (the difference is just the operand pruning at the
> moment of writing), similarly to what HIVE-26642 did for
> HiveReduceExpressionRule. Writing it here as a reminder, but it's recommended
> to tackle this in a separate ticket after verifying that is feasible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26789) Add UserName in CallerContext for queries
[ https://issues.apache.org/jira/browse/HIVE-26789?focusedWorklogId=830709=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830709 ] ASF GitHub Bot logged work on HIVE-26789: - Author: ASF GitHub Bot Created on: 02/Dec/22 18:20 Start Date: 02/Dec/22 18:20 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3813: URL: https://github.com/apache/hive/pull/3813#issuecomment-1335646946 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3813) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3813=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3813=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3813=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3813=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3813=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 830709) Time Spent: 2h 40m (was: 2.5h) > Add UserName in CallerContext for queries > - > > Key: HIVE-26789 > URL: https://issues.apache.org/jira/browse/HIVE-26789 > Project: Hive > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > HDFS Audit logs if impersonation is false, tracks only the Hive user in the > audit log, Can pass the actual user as part of the CallerContext, so that can > be logged as well for better tracking -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26799) Make authorizations on custom UDFs involved in tables/view configurable.
[
https://issues.apache.org/jira/browse/HIVE-26799?focusedWorklogId=830706=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830706
]
ASF GitHub Bot logged work on HIVE-26799:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 18:15
Start Date: 02/Dec/22 18:15
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera commented on code in PR #3821:
URL: https://github.com/apache/hive/pull/3821#discussion_r1038402915
##
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/command/CommandAuthorizerV2.java:
##
@@ -139,6 +139,13 @@ private static List
getHivePrivObjects(List Make authorizations on custom UDFs involved in tables/view configurable.
>
>
> Key: HIVE-26799
> URL: https://issues.apache.org/jira/browse/HIVE-26799
> Project: Hive
> Issue Type: New Feature
> Components: HiveServer2, Security
>Affects Versions: 4.0.0-alpha-2
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When Hive is using Ranger/Sentry as an authorization service, consider the
> following scenario.
> {code:java}
> > create table test_udf(st string); // privileged user operation
> > create function Udf_UPPER as 'openkb.hive.udf.MyUpper' using jar
> > 'hdfs:///tmp/MyUpperUDF-1.0.0.jar'; // privileged user operation
> > create view v1_udf as select udf_upper(st) from test_udf; // privileged
> > user operation
> //unprivileged user test_user is given select permissions on view v1_udf
> > select * from v1_udf; {code}
> It is expected that test_user needs to have select privilege on v1_udf and
> select permissions on udf_upper custom UDF in order to do a select query on
> view.
> This patch introduces a configuration
> "hive.security.authorization.functions.in.view"=false which disables
> authorization on views associated with views/tables during the select query.
> In this mode, only UDFs explicitly stated in the query would still be
> authorized as it is currently.
> The reason for making these custom UDFs associated with view/tables
> authorizable is that currently, test_user will need to be granted select
> permissions on the custom udf. and the test_user can use this UDF and query
> against any other table, which is a security concern.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26768) HPLSQL UDF is not working if it is applied on a column of type varchar/char/decimal in a table.
[
https://issues.apache.org/jira/browse/HIVE-26768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dayakar M updated HIVE-26768:
-
Fix Version/s: 4.0.0
> HPLSQL UDF is not working if it is applied on a column of type
> varchar/char/decimal in a table.
> ---
>
> Key: HIVE-26768
> URL: https://issues.apache.org/jira/browse/HIVE-26768
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Dayakar M
>Assignee: Dayakar M
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> HPLSQL UDF is not working if it is applied on a column of type
> varchar/char/decimal in a table.
> For example:
> {code:java}
> CREATE TABLE result (s varchar(20));
> INSERT INTO result VALUES('alice');
> INSERT INTO result VALUES('bob');
> CREATE FUNCTION hello(p string)
> RETURNS STRING
> BEGIN
> RETURN 'Hello, ' || p;
> END;
> SELECT hello(s) FROM result; {code}
>
> --> It should return below
> {code:java}
> ++
> | _c0 |
> ++
> | Hello, alice |
> | Hello, bob |
> ++
> {code}
>
> But actual result is
> {code:java}
> ++
> | _c0 |
> ++
> | Hello, |
> | Hello, |
> ++
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26754) Implement array_distinct UDF to return an array after removing duplicates in it
[
https://issues.apache.org/jira/browse/HIVE-26754?focusedWorklogId=830689=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830689
]
ASF GitHub Bot logged work on HIVE-26754:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 17:15
Start Date: 02/Dec/22 17:15
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3806:
URL: https://github.com/apache/hive/pull/3806#issuecomment-133464
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3806)
[](https://sonarcloud.io/project/issues?id=apache_hive=3806=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3806=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3806=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3806=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3806=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3806=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3806=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3806=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3806=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3806=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3806=false=CODE_SMELL)
[5 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3806=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3806=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3806=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 830689)
Time Spent: 3h 40m (was: 3.5h)
> Implement array_distinct UDF to return an array after removing duplicates in
> it
> ---
>
> Key: HIVE-26754
> URL: https://issues.apache.org/jira/browse/HIVE-26754
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> *array_distinct(array(obj1, obj2,...))* - The function returns an array of
> the same type as the input argument where all duplicate values have been
> removed.
> Example:
> > SELECT array_distinct(array('b', 'd', 'd', 'a')) FROM src LIMIT 1;
> ['a', 'b', 'c']
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26789) Add UserName in CallerContext for queries
[ https://issues.apache.org/jira/browse/HIVE-26789?focusedWorklogId=830671=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830671 ] ASF GitHub Bot logged work on HIVE-26789: - Author: ASF GitHub Bot Created on: 02/Dec/22 15:41 Start Date: 02/Dec/22 15:41 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3813: URL: https://github.com/apache/hive/pull/3813#issuecomment-1335438432 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3813) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3813=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3813=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3813=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=3813=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3813=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3813=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 830671) Time Spent: 2.5h (was: 2h 20m) > Add UserName in CallerContext for queries > - > > Key: HIVE-26789 > URL: https://issues.apache.org/jira/browse/HIVE-26789 > Project: Hive > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > HDFS Audit logs if impersonation is false, tracks only the Hive user in the > audit log, Can pass the actual user as part of the CallerContext, so that can > be logged as well for better tracking -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26806) Precommit tests in CI are timing out after HIVE-26796
[
https://issues.apache.org/jira/browse/HIVE-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642580#comment-17642580
]
Stamatis Zampetakis commented on HIVE-26806:
An attemt to fix the problem by increasing the SPLIT size from 20 to 22 was
committed to master:
https://github.com/apache/hive/commit/74729e0aa82951387dd5449862203a9a32282801.
However the initial Jenkins run for the respective master commit
(http://ci.hive.apache.org/job/hive-precommit/job/master/1508/parameters/)
didn't seem to pick up the changes for this specific parameter. We stopped the
run and manually re-launched
http://ci.hive.apache.org/job/hive-precommit/job/master/1510/parameters/ and
now we are waiting for the results.
> Precommit tests in CI are timing out after HIVE-26796
> -
>
> Key: HIVE-26806
> URL: https://issues.apache.org/jira/browse/HIVE-26806
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> http://ci.hive.apache.org/job/hive-precommit/job/master/1506/
> {noformat}
> ancelling nested steps due to timeout
> 15:22:08 Sending interrupt signal to process
> 15:22:08 Killing processes
> 15:22:09 kill finished with exit code 0
> 15:22:19 Terminated
> 15:22:19 script returned exit code 143
> [Pipeline] }
> [Pipeline] // withEnv
> [Pipeline] }
> 15:22:19 Deleting 1 temporary files
> [Pipeline] // configFileProvider
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (PostProcess)
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] junit
> 15:22:25 Recording test results
> 15:22:32 [Checks API] No suitable checks publisher found.
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] }
> [Pipeline] // container
> [Pipeline] }
> [Pipeline] // node
> [Pipeline] }
> [Pipeline] // timeout
> [Pipeline] }
> [Pipeline] // podTemplate
> [Pipeline] }
> 15:22:32 Failed in branch split-01
> [Pipeline] // parallel
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (Archive)
> [Pipeline] podTemplate
> [Pipeline] {
> [Pipeline] timeout
> 15:22:33 Timeout set to expire in 6 hr 0 min
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26692) Check for the expected thrift version before compiling
[ https://issues.apache.org/jira/browse/HIVE-26692?focusedWorklogId=830666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830666 ] ASF GitHub Bot logged work on HIVE-26692: - Author: ASF GitHub Bot Created on: 02/Dec/22 15:21 Start Date: 02/Dec/22 15:21 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3820: URL: https://github.com/apache/hive/pull/3820#issuecomment-1335408246 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3820) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3820=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3820=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3820=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3820=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3820=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3820=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 830666) Time Spent: 1h 50m (was: 1h 40m) > Check for the expected thrift version before compiling > -- > > Key: HIVE-26692 > URL: https://issues.apache.org/jira/browse/HIVE-26692 > Project: Hive > Issue Type: Task > Components: Thrift API >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > At the moment we don't check for the thrift version before launching thrift, > the error messages are often cryptic upon mismatches. > An explicit check with a clear error message would be nice, like what parquet > does: > [https://github.com/apache/parquet-mr/blob/master/parquet-thrift/pom.xml#L247-L268] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26806) Precommit tests in CI are timing out after HIVE-26796
[
https://issues.apache.org/jira/browse/HIVE-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642558#comment-17642558
]
Alessandro Solimando commented on HIVE-26806:
-
In case you have an existing open PR suffering form this and you don't want to
rebase, if you have permission to run Jenkins' jobs you just change the default
split value to 22 and re-run, HTH
> Precommit tests in CI are timing out after HIVE-26796
> -
>
> Key: HIVE-26806
> URL: https://issues.apache.org/jira/browse/HIVE-26806
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> http://ci.hive.apache.org/job/hive-precommit/job/master/1506/
> {noformat}
> ancelling nested steps due to timeout
> 15:22:08 Sending interrupt signal to process
> 15:22:08 Killing processes
> 15:22:09 kill finished with exit code 0
> 15:22:19 Terminated
> 15:22:19 script returned exit code 143
> [Pipeline] }
> [Pipeline] // withEnv
> [Pipeline] }
> 15:22:19 Deleting 1 temporary files
> [Pipeline] // configFileProvider
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (PostProcess)
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] junit
> 15:22:25 Recording test results
> 15:22:32 [Checks API] No suitable checks publisher found.
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] }
> [Pipeline] // container
> [Pipeline] }
> [Pipeline] // node
> [Pipeline] }
> [Pipeline] // timeout
> [Pipeline] }
> [Pipeline] // podTemplate
> [Pipeline] }
> 15:22:32 Failed in branch split-01
> [Pipeline] // parallel
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (Archive)
> [Pipeline] podTemplate
> [Pipeline] {
> [Pipeline] timeout
> 15:22:33 Timeout set to expire in 6 hr 0 min
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26806) Precommit tests in CI are timing out after HIVE-26796
[
https://issues.apache.org/jira/browse/HIVE-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642552#comment-17642552
]
Stamatis Zampetakis commented on HIVE-26806:
The fact that the tests are taking too much time to execute could be due to a
bad allocation to buckets from [Jenkins Parallel Test Executor
Plugin|https://www.jenkins.io/doc/pipeline/steps/parallel-test-executor/#splittests-split-test-runs].
The problem is probably a result of HIVE-26796 that re-enabled ~1K more tests
that were not distributed evenly among splits.
> Precommit tests in CI are timing out after HIVE-26796
> -
>
> Key: HIVE-26806
> URL: https://issues.apache.org/jira/browse/HIVE-26806
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> http://ci.hive.apache.org/job/hive-precommit/job/master/1506/
> {noformat}
> ancelling nested steps due to timeout
> 15:22:08 Sending interrupt signal to process
> 15:22:08 Killing processes
> 15:22:09 kill finished with exit code 0
> 15:22:19 Terminated
> 15:22:19 script returned exit code 143
> [Pipeline] }
> [Pipeline] // withEnv
> [Pipeline] }
> 15:22:19 Deleting 1 temporary files
> [Pipeline] // configFileProvider
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (PostProcess)
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] junit
> 15:22:25 Recording test results
> 15:22:32 [Checks API] No suitable checks publisher found.
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] }
> [Pipeline] // container
> [Pipeline] }
> [Pipeline] // node
> [Pipeline] }
> [Pipeline] // timeout
> [Pipeline] }
> [Pipeline] // podTemplate
> [Pipeline] }
> 15:22:32 Failed in branch split-01
> [Pipeline] // parallel
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (Archive)
> [Pipeline] podTemplate
> [Pipeline] {
> [Pipeline] timeout
> 15:22:33 Timeout set to expire in 6 hr 0 min
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26806) Precommit tests in CI are timing out after HIVE-26796
[
https://issues.apache.org/jira/browse/HIVE-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis reassigned HIVE-26806:
--
> Precommit tests in CI are timing out after HIVE-26796
> -
>
> Key: HIVE-26806
> URL: https://issues.apache.org/jira/browse/HIVE-26806
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> http://ci.hive.apache.org/job/hive-precommit/job/master/1506/
> {noformat}
> ancelling nested steps due to timeout
> 15:22:08 Sending interrupt signal to process
> 15:22:08 Killing processes
> 15:22:09 kill finished with exit code 0
> 15:22:19 Terminated
> 15:22:19 script returned exit code 143
> [Pipeline] }
> [Pipeline] // withEnv
> [Pipeline] }
> 15:22:19 Deleting 1 temporary files
> [Pipeline] // configFileProvider
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (PostProcess)
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] sh
> [Pipeline] junit
> 15:22:25 Recording test results
> 15:22:32 [Checks API] No suitable checks publisher found.
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] }
> [Pipeline] // container
> [Pipeline] }
> [Pipeline] // node
> [Pipeline] }
> [Pipeline] // timeout
> [Pipeline] }
> [Pipeline] // podTemplate
> [Pipeline] }
> 15:22:32 Failed in branch split-01
> [Pipeline] // parallel
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] stage
> [Pipeline] { (Archive)
> [Pipeline] podTemplate
> [Pipeline] {
> [Pipeline] timeout
> 15:22:33 Timeout set to expire in 6 hr 0 min
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26770) Make "end of loop" compaction logs appear more selectively
[
https://issues.apache.org/jira/browse/HIVE-26770?focusedWorklogId=830653=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830653
]
ASF GitHub Bot logged work on HIVE-26770:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 14:06
Start Date: 02/Dec/22 14:06
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3803:
URL: https://github.com/apache/hive/pull/3803#issuecomment-1335278590
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3803)
[](https://sonarcloud.io/project/issues?id=apache_hive=3803=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3803=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3803=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3803=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3803=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3803=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3803=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3803=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3803=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3803=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3803=false=CODE_SMELL)
[10 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3803=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3803=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3803=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 830653)
Time Spent: 4h 20m (was: 4h 10m)
> Make "end of loop" compaction logs appear more selectively
> --
>
> Key: HIVE-26770
> URL: https://issues.apache.org/jira/browse/HIVE-26770
> Project: Hive
> Issue Type: Improvement
>Reporter: Akshat Mathur
>Assignee: Akshat Mathur
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Currently Initiator, Worker, and Cleaner threads log something like "finished
> one loop" on INFO level.
> This is useful to figure out if one of these threads is taking too long to
> finish a loop, but expensive in general.
>
> Suggested Time: 20mins
> Logging this should be changed in the following way
> # If loop finished within a predefined amount of time, level should be DEBUG
> and message should look like: *Initiator loop took \{ellapsedTime} seconds to
> finish.*
> # If loop ran longer than this predefined amount, level should be WARN and
> message should look like: *Possible Initiator slowdown, loop took
> \{ellapsedTime} seconds to finish.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26767) Support for custom RDBMS is broken
[
https://issues.apache.org/jira/browse/HIVE-26767?focusedWorklogId=830645=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830645
]
ASF GitHub Bot logged work on HIVE-26767:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 13:48
Start Date: 02/Dec/22 13:48
Worklog Time Spent: 10m
Work Description: tthorpeIBM commented on PR #3799:
URL: https://github.com/apache/hive/pull/3799#issuecomment-1335256121
@saihemanth-cloudera Thank you!
Issue Time Tracking
---
Worklog Id: (was: 830645)
Time Spent: 2h 20m (was: 2h 10m)
> Support for custom RDBMS is broken
> --
>
> Key: HIVE-26767
> URL: https://issues.apache.org/jira/browse/HIVE-26767
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Tim Thorpe
>Assignee: Tim Thorpe
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> HIVE-24120 introduced code to support custom RDBMS.
> DatabaseProduct.getDbType(String productName) will return *DbType.UNDEFINED*
> for anything other than the hardcoded/internally supported database types.
> When initializing DatabaseProduct with an external/custom RDBMS, it follows
> this logic:
>
> boolean isExternal = MetastoreConf.getBoolVar(conf,
> ConfVars.USE_CUSTOM_RDBMS);
> if (isExternal) {
> // The DatabaseProduct will be created by instantiating an external
> class via
> // reflection. The external class can override any method in the
> current class
> String className = MetastoreConf.getVar(conf,
> ConfVars.CUSTOM_RDBMS_CLASSNAME);
> if (className != null) {
> try {
> theDatabaseProduct = (DatabaseProduct)
> ReflectionUtils.newInstance(Class.forName(className), conf);
> LOG.info(String.format("Using custom RDBMS %s", className));
> dbt = DbType.CUSTOM;
> These 2 database types (DbType.UNDEFINED, DbType.CUSTOM) are then compared to
> each other to make sure they are the same.
>
> Preconditions.checkState(theDatabaseProduct.dbType == getDbType(productName));
>
> [https://github.com/gatorblue/hive/blob/3a65c6cf9cb552e7c34bfb449a419abfde0a58b6/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java#L80]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26801) Query based compaction fails on tables having columns with keywords(i.e. row in this case)
[
https://issues.apache.org/jira/browse/HIVE-26801?focusedWorklogId=830630=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830630
]
ASF GitHub Bot logged work on HIVE-26801:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 12:48
Start Date: 02/Dec/22 12:48
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3828:
URL: https://github.com/apache/hive/pull/3828#issuecomment-1335180838
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3828)
[](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3828=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3828=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3828=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3828=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3828=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3828=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 830630)
Time Spent: 20m (was: 10m)
> Query based compaction fails on tables having columns with keywords(i.e. row
> in this case)
> --
>
> Key: HIVE-26801
> URL: https://issues.apache.org/jira/browse/HIVE-26801
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Gopinath
>Assignee: Gopinath
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Query-based compaction fails on tables having columns with keywords(i.e. row
> in this case) for columns. The compaction fails while running insert into
> statement as it does not quote the columns correctly.
> Below are the steps to reproduce the issue.
> {noformat}
> CREATE TABLE aggregated_data(`sessionid` string,`row` int,`timeofoccurrence`
> bigint);
> insert into table aggregated_data values ("abcd",300,211);
> insert into table aggregated_data values ("abcd",300,211);
> alter table aggregated_data compact 'MAJOR' and wait;{noformat}
> Error -
> {noformat}
> 2022-11-30 13:04:33,309 INFO
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor: [repro894918]:
> Running major compaction via query: INSERT into table
> default_tmp_compactor_aggregated_data_1669813472898 select
> validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
> ROW__ID.writeId,
[jira] [Commented] (HIVE-26158) TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after rename table
[
https://issues.apache.org/jira/browse/HIVE-26158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642475#comment-17642475
]
LiPenglin commented on HIVE-26158:
--
Hi [~kgyrtkirk] and [~sanguines]! I didn't find any more information about
'TRANSLATED_TO_EXTERNAL', so I ask here.
I just want to make sure that location doesn't change when I rename a table, so
what I'm doing right now is:
1 set 'EXTERNAL'='TRUE'
2 set 'TRANSLATED_TO_EXTERNAL'='FALSE'
3 set 'external.table.purge'='false'
Does it make sense for me to do this? And how can I understand
'TRANSLATED_TO_EXTERNAL' better?
Regards!
> TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after
> rename table
> --
>
> Key: HIVE-26158
> URL: https://issues.apache.org/jira/browse/HIVE-26158
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: tanghui
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: metastore_translator, pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> After the patch is updated, the partition table location and hdfs data
> directory are displayed normally, but the partition location of the table in
> the SDS in the Hive metabase is still displayed as the location of the old
> table, resulting in no data in the query partition.
>
> in beeline:
>
> set hive.create.as.external.legacy=true;
> CREATE TABLE part_test(
> c1 string
> ,c2 string
> )PARTITIONED BY (dat string)
> insert into part_test values ("11","th","20220101")
> insert into part_test values ("22","th","20220102")
> alter table part_test rename to part_test11;
> --this result is null.
> select * from part_test11 where dat="20220101";
> ||part_test.c1||part_test.c2||part_test.dat||
> | | | |
> -
> SDS in the Hive metabase:
> select SDS.LOCATION from TBLS,SDS where TBLS.TBL_NAME="part_test11" AND
> TBLS.TBL_ID=SDS.CD_ID;
> ---
> |*LOCATION*|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test11|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220101|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220102|
> ---
>
> We need to modify the partition location of the table in SDS to ensure that
> the query results are normal
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26796) All tests in hive-unit module are skipped silently
[
https://issues.apache.org/jira/browse/HIVE-26796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642461#comment-17642461
]
Denys Kuzmenko commented on HIVE-26796:
---
now it's taking forever to execute:
http://ci.hive.apache.org/job/hive-precommit/job/master/1506/
> All tests in hive-unit module are skipped silently
> --
>
> Key: HIVE-26796
> URL: https://issues.apache.org/jira/browse/HIVE-26796
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In current master (7207a62def246b3290f1ece529e65b79012a3578) the tests in
> hive-unit module are not running.
> {noformat}
> $ cd itests/hive-unit && mvn test
> [INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ hive-it-unit
> ---
> [INFO]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26754) Implement array_distinct UDF to return an array after removing duplicates in it
[
https://issues.apache.org/jira/browse/HIVE-26754?focusedWorklogId=830621=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830621
]
ASF GitHub Bot logged work on HIVE-26754:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 11:37
Start Date: 02/Dec/22 11:37
Worklog Time Spent: 10m
Work Description: tarak271 commented on code in PR #3806:
URL: https://github.com/apache/hive/pull/3806#discussion_r1038057429
##
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArrayDistinct.java:
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.udf.generic;
+
+import org.apache.hadoop.hive.ql.exec.Description;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Generic UDF for distinct array
+ * ARRAY_DISTINCT(array(obj1, obj2, obj3...)).
+ *
+ * @see org.apache.hadoop.hive.ql.udf.generic.GenericUDF
+ */
+@Description(name = "array_distinct",
+value = "_FUNC_(array(obj1, obj2,...)) - "
++ "The function returns an array of the same type as the input
array with distinct values.",
+extended = "Example:\n"
++ " > SELECT _FUNC_(array('b', 'd', 'd', 'a')) FROM src LIMIT
1;\n"
++ " ['b', 'd', 'a']")
+public class GenericUDFArrayDistinct extends AbstractGenericUDFArrayBase {
+
+public GenericUDFArrayDistinct() {
+super("ARRAY_DISTINCT", 1, 1, ObjectInspector.Category.LIST);
+}
+
+@Override
+public Object evaluate(DeferredObject[] arguments) throws HiveException {
+
+Object array = arguments[ARRAY_IDX].get();
+
+if (arrayOI.getListLength(array) <= 0) {
+return new ArrayList();
+}
Review Comment:
And added another Test class to simulate null Object, TestGenericUDFArray
which can be reused across Array functions
Issue Time Tracking
---
Worklog Id: (was: 830621)
Time Spent: 3.5h (was: 3h 20m)
> Implement array_distinct UDF to return an array after removing duplicates in
> it
> ---
>
> Key: HIVE-26754
> URL: https://issues.apache.org/jira/browse/HIVE-26754
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> *array_distinct(array(obj1, obj2,...))* - The function returns an array of
> the same type as the input argument where all duplicate values have been
> removed.
> Example:
> > SELECT array_distinct(array('b', 'd', 'd', 'a')) FROM src LIMIT 1;
> ['a', 'b', 'c']
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26754) Implement array_distinct UDF to return an array after removing duplicates in it
[
https://issues.apache.org/jira/browse/HIVE-26754?focusedWorklogId=830619=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830619
]
ASF GitHub Bot logged work on HIVE-26754:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 11:35
Start Date: 02/Dec/22 11:35
Worklog Time Spent: 10m
Work Description: tarak271 commented on code in PR #3806:
URL: https://github.com/apache/hive/pull/3806#discussion_r1038055787
##
ql/src/test/queries/clientpositive/udf_array_distinct.q:
##
@@ -0,0 +1,33 @@
+--! qt:dataset:src
+set hive.fetch.task.conversion=more;
Review Comment:
Added flag
##
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFArrayBase.java:
##
@@ -106,4 +106,4 @@ ObjectInspector initListOI(ObjectInspector[] arguments) {
return
ObjectInspectorFactory.getStandardListObjectInspector(initOI(arguments));
}
-}
+}
Review Comment:
Done
Issue Time Tracking
---
Worklog Id: (was: 830619)
Time Spent: 3h 10m (was: 3h)
> Implement array_distinct UDF to return an array after removing duplicates in
> it
> ---
>
> Key: HIVE-26754
> URL: https://issues.apache.org/jira/browse/HIVE-26754
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> *array_distinct(array(obj1, obj2,...))* - The function returns an array of
> the same type as the input argument where all duplicate values have been
> removed.
> Example:
> > SELECT array_distinct(array('b', 'd', 'd', 'a')) FROM src LIMIT 1;
> ['a', 'b', 'c']
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26754) Implement array_distinct UDF to return an array after removing duplicates in it
[
https://issues.apache.org/jira/browse/HIVE-26754?focusedWorklogId=830618=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830618
]
ASF GitHub Bot logged work on HIVE-26754:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 11:35
Start Date: 02/Dec/22 11:35
Worklog Time Spent: 10m
Work Description: tarak271 commented on code in PR #3806:
URL: https://github.com/apache/hive/pull/3806#discussion_r1038055541
##
ql/src/test/queries/clientpositive/udf_array_distinct.q:
##
@@ -0,0 +1,33 @@
+--! qt:dataset:src
+set hive.fetch.task.conversion=more;
+
+DESCRIBE FUNCTION array_distinct;
+DESCRIBE FUNCTION EXTENDED array_distinct;
+
+
Issue Time Tracking
---
Worklog Id: (was: 830618)
Time Spent: 3h (was: 2h 50m)
> Implement array_distinct UDF to return an array after removing duplicates in
> it
> ---
>
> Key: HIVE-26754
> URL: https://issues.apache.org/jira/browse/HIVE-26754
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> *array_distinct(array(obj1, obj2,...))* - The function returns an array of
> the same type as the input argument where all duplicate values have been
> removed.
> Example:
> > SELECT array_distinct(array('b', 'd', 'd', 'a')) FROM src LIMIT 1;
> ['a', 'b', 'c']
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26754) Implement array_distinct UDF to return an array after removing duplicates in it
[
https://issues.apache.org/jira/browse/HIVE-26754?focusedWorklogId=830620=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830620
]
ASF GitHub Bot logged work on HIVE-26754:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 11:35
Start Date: 02/Dec/22 11:35
Worklog Time Spent: 10m
Work Description: tarak271 commented on code in PR #3806:
URL: https://github.com/apache/hive/pull/3806#discussion_r1038056071
##
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArrayDistinct.java:
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.udf.generic;
+
+import org.apache.hadoop.hive.ql.exec.Description;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Generic UDF for distinct array
+ * ARRAY_DISTINCT(array(obj1, obj2, obj3...)).
+ *
+ * @see org.apache.hadoop.hive.ql.udf.generic.GenericUDF
+ */
+@Description(name = "array_distinct",
+value = "_FUNC_(array(obj1, obj2,...)) - "
++ "The function returns an array of the same type as the input
array with distinct values.",
+extended = "Example:\n"
++ " > SELECT _FUNC_(array('b', 'd', 'd', 'a')) FROM src LIMIT
1;\n"
++ " ['b', 'd', 'a']")
+public class GenericUDFArrayDistinct extends AbstractGenericUDFArrayBase {
+
+public GenericUDFArrayDistinct() {
+super("ARRAY_DISTINCT", 1, 1, ObjectInspector.Category.LIST);
+}
+
+@Override
+public Object evaluate(DeferredObject[] arguments) throws HiveException {
+
+Object array = arguments[ARRAY_IDX].get();
+
+if (arrayOI.getListLength(array) <= 0) {
+return new ArrayList();
+}
Review Comment:
Changed as recommended
Issue Time Tracking
---
Worklog Id: (was: 830620)
Time Spent: 3h 20m (was: 3h 10m)
> Implement array_distinct UDF to return an array after removing duplicates in
> it
> ---
>
> Key: HIVE-26754
> URL: https://issues.apache.org/jira/browse/HIVE-26754
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> *array_distinct(array(obj1, obj2,...))* - The function returns an array of
> the same type as the input argument where all duplicate values have been
> removed.
> Example:
> > SELECT array_distinct(array('b', 'd', 'd', 'a')) FROM src LIMIT 1;
> ['a', 'b', 'c']
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26801) Query based compaction fails on tables having columns with keywords(i.e. row in this case)
[
https://issues.apache.org/jira/browse/HIVE-26801?focusedWorklogId=830615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830615
]
ASF GitHub Bot logged work on HIVE-26801:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 11:09
Start Date: 02/Dec/22 11:09
Worklog Time Spent: 10m
Work Description: ggangadharan opened a new pull request, #3828:
URL: https://github.com/apache/hive/pull/3828
### What changes were proposed in this pull request?
Query-based compaction fails on tables having columns with keywords(i.e. row
in this case) for columns. The compaction fails while running insert into
statement as it does not quote the columns correctly.
### Why are the changes needed?
This fixes the current bug with Query based compaction
### Does this PR introduce _any_ user-facing change?
NO
### How was this patch tested?
Currently tested manually, and added a UT to
TestCrudCompactorOnTez#testCompactionShouldNotFailOnKeywordField
Issue Time Tracking
---
Worklog Id: (was: 830615)
Remaining Estimate: 0h
Time Spent: 10m
> Query based compaction fails on tables having columns with keywords(i.e. row
> in this case)
> --
>
> Key: HIVE-26801
> URL: https://issues.apache.org/jira/browse/HIVE-26801
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Gopinath
>Assignee: Gopinath
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Query-based compaction fails on tables having columns with keywords(i.e. row
> in this case) for columns. The compaction fails while running insert into
> statement as it does not quote the columns correctly.
> Below are the steps to reproduce the issue.
> {noformat}
> CREATE TABLE aggregated_data(`sessionid` string,`row` int,`timeofoccurrence`
> bigint);
> insert into table aggregated_data values ("abcd",300,211);
> insert into table aggregated_data values ("abcd",300,211);
> alter table aggregated_data compact 'MAJOR' and wait;{noformat}
> Error -
> {noformat}
> 2022-11-30 13:04:33,309 INFO
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor: [repro894918]:
> Running major compaction via query: INSERT into table
> default_tmp_compactor_aggregated_data_1669813472898 select
> validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
> ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
> NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
> timeofoccurrence) from default.aggregated_data
> 2022-11-30 13:04:33,309 INFO org.apache.hadoop.hive.ql.Driver:
> [repro894918]: Compiling
> command(queryId=hive_20221130130433_de2a8b2d-f993-44e5-8aeb-decba3342a85):
> INSERT into table default_tmp_compactor_aggregated_data_1669813472898 select
> validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
> ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
> NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
> timeofoccurrence) from default.aggregated_data
> 2022-11-30 13:04:33,314 ERROR org.apache.hadoop.hive.ql.Driver:
> [repro894918]: FAILED: ParseException line 1:277 cannot recognize input near
> 'row' ',' ''timeofoccurrence'' in select expression
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:277 cannot recognize
> input near 'row' ',' ''timeofoccurrence'' in select expression
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:128)
> at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:82)
> at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74)
> at org.apache.hadoop.hive.ql.Compiler.parse(Compiler.java:173)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:102)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:196)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:615)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:673)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:505)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:494)
> at
> org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:70)
> at
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:133)
> at
> org.apache.hadoop.hive.ql.txn.compactor.MajorQueryCompactor.runCompaction(MajorQueryCompactor.java:63)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.findNextCompactionAndExecute(Worker.java:562)
> at
>
[jira] [Updated] (HIVE-26801) Query based compaction fails on tables having columns with keywords(i.e. row in this case)
[
https://issues.apache.org/jira/browse/HIVE-26801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26801:
--
Labels: pull-request-available (was: )
> Query based compaction fails on tables having columns with keywords(i.e. row
> in this case)
> --
>
> Key: HIVE-26801
> URL: https://issues.apache.org/jira/browse/HIVE-26801
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Gopinath
>Assignee: Gopinath
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Query-based compaction fails on tables having columns with keywords(i.e. row
> in this case) for columns. The compaction fails while running insert into
> statement as it does not quote the columns correctly.
> Below are the steps to reproduce the issue.
> {noformat}
> CREATE TABLE aggregated_data(`sessionid` string,`row` int,`timeofoccurrence`
> bigint);
> insert into table aggregated_data values ("abcd",300,211);
> insert into table aggregated_data values ("abcd",300,211);
> alter table aggregated_data compact 'MAJOR' and wait;{noformat}
> Error -
> {noformat}
> 2022-11-30 13:04:33,309 INFO
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor: [repro894918]:
> Running major compaction via query: INSERT into table
> default_tmp_compactor_aggregated_data_1669813472898 select
> validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
> ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
> NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
> timeofoccurrence) from default.aggregated_data
> 2022-11-30 13:04:33,309 INFO org.apache.hadoop.hive.ql.Driver:
> [repro894918]: Compiling
> command(queryId=hive_20221130130433_de2a8b2d-f993-44e5-8aeb-decba3342a85):
> INSERT into table default_tmp_compactor_aggregated_data_1669813472898 select
> validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
> ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
> NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
> timeofoccurrence) from default.aggregated_data
> 2022-11-30 13:04:33,314 ERROR org.apache.hadoop.hive.ql.Driver:
> [repro894918]: FAILED: ParseException line 1:277 cannot recognize input near
> 'row' ',' ''timeofoccurrence'' in select expression
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:277 cannot recognize
> input near 'row' ',' ''timeofoccurrence'' in select expression
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:128)
> at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:82)
> at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74)
> at org.apache.hadoop.hive.ql.Compiler.parse(Compiler.java:173)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:102)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:196)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:615)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:673)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:505)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:494)
> at
> org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:70)
> at
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:133)
> at
> org.apache.hadoop.hive.ql.txn.compactor.MajorQueryCompactor.runCompaction(MajorQueryCompactor.java:63)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.findNextCompactionAndExecute(Worker.java:562)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.lambda$run$0(Worker.java:113)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26716) Query based Rebalance compaction on full acid tables
[ https://issues.apache.org/jira/browse/HIVE-26716?focusedWorklogId=830606=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830606 ] ASF GitHub Bot logged work on HIVE-26716: - Author: ASF GitHub Bot Created on: 02/Dec/22 10:21 Start Date: 02/Dec/22 10:21 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3746: URL: https://github.com/apache/hive/pull/3746#issuecomment-1335038886 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3746) [](https://sonarcloud.io/project/issues?id=apache_hive=3746=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3746=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3746=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3746=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3746=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3746=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3746=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3746=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3746=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3746=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3746=false=CODE_SMELL) [84 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3746=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3746=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3746=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 830606) Time Spent: 17h 10m (was: 17h) > Query based Rebalance compaction on full acid tables > > > Key: HIVE-26716 > URL: https://issues.apache.org/jira/browse/HIVE-26716 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: ACID, compaction, pull-request-available > Time Spent: 17h 10m > Remaining Estimate: 0h > > Support rebalancing compaction on fully ACID tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26805) Cancel ongoing/working compaction requests
[ https://issues.apache.org/jira/browse/HIVE-26805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KIRTI RUGE reassigned HIVE-26805: - Assignee: KIRTI RUGE > Cancel ongoing/working compaction requests > -- > > Key: HIVE-26805 > URL: https://issues.apache.org/jira/browse/HIVE-26805 > Project: Hive > Issue Type: New Feature >Reporter: KIRTI RUGE >Assignee: KIRTI RUGE >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26803) Ability to cancel compactions
[ https://issues.apache.org/jira/browse/HIVE-26803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KIRTI RUGE updated HIVE-26803: -- Description: This has to take care of below tasks: # Cancel compaction in initiated state # Cancel ongoing/working comacions # Gracefully handle compaction request when HS2 shuts down > Ability to cancel compactions > - > > Key: HIVE-26803 > URL: https://issues.apache.org/jira/browse/HIVE-26803 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: KIRTI RUGE >Assignee: KIRTI RUGE >Priority: Major > > This has to take care of below tasks: > # Cancel compaction in initiated state > # Cancel ongoing/working comacions > # Gracefully handle compaction request when HS2 shuts down -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26804) Cancel Compactions in initiated state
[ https://issues.apache.org/jira/browse/HIVE-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KIRTI RUGE reassigned HIVE-26804: - Assignee: KIRTI RUGE > Cancel Compactions in initiated state > - > > Key: HIVE-26804 > URL: https://issues.apache.org/jira/browse/HIVE-26804 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: KIRTI RUGE >Assignee: KIRTI RUGE >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26803) Ability to cancel compactions
[ https://issues.apache.org/jira/browse/HIVE-26803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KIRTI RUGE reassigned HIVE-26803: - Assignee: KIRTI RUGE > Ability to cancel compactions > - > > Key: HIVE-26803 > URL: https://issues.apache.org/jira/browse/HIVE-26803 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: KIRTI RUGE >Assignee: KIRTI RUGE >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26802) Create qtest running QB compaction queries
[ https://issues.apache.org/jira/browse/HIVE-26802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Rátkai reassigned HIVE-26802: Assignee: Zoltán Rátkai > Create qtest running QB compaction queries > -- > > Key: HIVE-26802 > URL: https://issues.apache.org/jira/browse/HIVE-26802 > Project: Hive > Issue Type: Improvement >Reporter: Zoltán Rátkai >Assignee: Zoltán Rátkai >Priority: Minor > > Create a qtest that runs the queries that query-based compaction runs. > Not so much to check for correct data but more to check the query plans, to > simplify tracing changes in compilation that might affect QB compaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26793) Create a new configuration to override "no compaction" for tables
[
https://issues.apache.org/jira/browse/HIVE-26793?focusedWorklogId=830579=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830579
]
ASF GitHub Bot logged work on HIVE-26793:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 08:54
Start Date: 02/Dec/22 08:54
Worklog Time Spent: 10m
Work Description: kokila-19 commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1037914520
##
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics2.java:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.hadoop.hive.metastore.HMSMetricsListener;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.metrics.Metrics;
+import org.apache.hadoop.hive.metastore.metrics.MetricsConstants;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+public class TestCompactionMetrics2 extends CompactorOnTezTest {
Review Comment:
But I see that there is a file in hive-unit-> TestCleanerWithReplication
extending hive-exec -> CompactorTest
Issue Time Tracking
---
Worklog Id: (was: 830579)
Time Spent: 1h 10m (was: 1h)
> Create a new configuration to override "no compaction" for tables
> -
>
> Key: HIVE-26793
> URL: https://issues.apache.org/jira/browse/HIVE-26793
> Project: Hive
> Issue Type: Improvement
>Reporter: Kokila N
>Assignee: Kokila N
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently a simple user can create a table with
> {color:#6a8759}no_auto_compaction=true{color} table property and create an
> aborted write transaction writing to this table. This way a malicious user
> can prevent cleaning up data for the aborted transaction, creating
> performance degradation.
> This configuration should be allowed to overridden on a database level:
> adding {color:#6a8759}no_auto_compaction=false{color} should override the
> table level setting forcing the initiator to schedule compaction for all
> tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26793) Create a new configuration to override "no compaction" for tables
[
https://issues.apache.org/jira/browse/HIVE-26793?focusedWorklogId=830578=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830578
]
ASF GitHub Bot logged work on HIVE-26793:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 08:53
Start Date: 02/Dec/22 08:53
Worklog Time Spent: 10m
Work Description: kokila-19 commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1037914520
##
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics2.java:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.hadoop.hive.metastore.HMSMetricsListener;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.metrics.Metrics;
+import org.apache.hadoop.hive.metastore.metrics.MetricsConstants;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+public class TestCompactionMetrics2 extends CompactorOnTezTest {
Review Comment:
But I see that there is a file in hive-it-util -> TestCleanerWithReplication
extending hive-exec -> CompactorTest
Issue Time Tracking
---
Worklog Id: (was: 830578)
Time Spent: 1h (was: 50m)
> Create a new configuration to override "no compaction" for tables
> -
>
> Key: HIVE-26793
> URL: https://issues.apache.org/jira/browse/HIVE-26793
> Project: Hive
> Issue Type: Improvement
>Reporter: Kokila N
>Assignee: Kokila N
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently a simple user can create a table with
> {color:#6a8759}no_auto_compaction=true{color} table property and create an
> aborted write transaction writing to this table. This way a malicious user
> can prevent cleaning up data for the aborted transaction, creating
> performance degradation.
> This configuration should be allowed to overridden on a database level:
> adding {color:#6a8759}no_auto_compaction=false{color} should override the
> table level setting forcing the initiator to schedule compaction for all
> tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26801) Query based compaction fails on tables having columns with keywords(i.e. row in this case)
[
https://issues.apache.org/jira/browse/HIVE-26801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopinath updated HIVE-26801:
Component/s: Hive
> Query based compaction fails on tables having columns with keywords(i.e. row
> in this case)
> --
>
> Key: HIVE-26801
> URL: https://issues.apache.org/jira/browse/HIVE-26801
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Gopinath
>Assignee: Gopinath
>Priority: Major
>
> Query-based compaction fails on tables having columns with keywords(i.e. row
> in this case) for columns. The compaction fails while running insert into
> statement as it does not quote the columns correctly.
> Below are the steps to reproduce the issue.
> {noformat}
> CREATE TABLE aggregated_data(`sessionid` string,`row` int,`timeofoccurrence`
> bigint);
> insert into table aggregated_data values ("abcd",300,211);
> insert into table aggregated_data values ("abcd",300,211);
> alter table aggregated_data compact 'MAJOR' and wait;{noformat}
> Error -
> {noformat}
> 2022-11-30 13:04:33,309 INFO
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor: [repro894918]:
> Running major compaction via query: INSERT into table
> default_tmp_compactor_aggregated_data_1669813472898 select
> validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
> ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
> NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
> timeofoccurrence) from default.aggregated_data
> 2022-11-30 13:04:33,309 INFO org.apache.hadoop.hive.ql.Driver:
> [repro894918]: Compiling
> command(queryId=hive_20221130130433_de2a8b2d-f993-44e5-8aeb-decba3342a85):
> INSERT into table default_tmp_compactor_aggregated_data_1669813472898 select
> validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
> ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
> NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
> timeofoccurrence) from default.aggregated_data
> 2022-11-30 13:04:33,314 ERROR org.apache.hadoop.hive.ql.Driver:
> [repro894918]: FAILED: ParseException line 1:277 cannot recognize input near
> 'row' ',' ''timeofoccurrence'' in select expression
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:277 cannot recognize
> input near 'row' ',' ''timeofoccurrence'' in select expression
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:128)
> at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:82)
> at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74)
> at org.apache.hadoop.hive.ql.Compiler.parse(Compiler.java:173)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:102)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:196)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:615)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:673)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:505)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:494)
> at
> org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:70)
> at
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:133)
> at
> org.apache.hadoop.hive.ql.txn.compactor.MajorQueryCompactor.runCompaction(MajorQueryCompactor.java:63)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.findNextCompactionAndExecute(Worker.java:562)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.lambda$run$0(Worker.java:113)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26801) Query based compaction fails on tables having columns with keywords(i.e. row in this case)
[
https://issues.apache.org/jira/browse/HIVE-26801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopinath updated HIVE-26801:
Description:
Query-based compaction fails on tables having columns with keywords(i.e. row in
this case) for columns. The compaction fails while running insert into
statement as it does not quote the columns correctly.
Below are the steps to reproduce the issue.
{noformat}
CREATE TABLE aggregated_data(`sessionid` string,`row` int,`timeofoccurrence`
bigint);
insert into table aggregated_data values ("abcd",300,211);
insert into table aggregated_data values ("abcd",300,211);
alter table aggregated_data compact 'MAJOR' and wait;{noformat}
Error -
{noformat}
2022-11-30 13:04:33,309 INFO
org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor: [repro894918]: Running
major compaction via query: INSERT into table
default_tmp_compactor_aggregated_data_1669813472898 select
validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
timeofoccurrence) from default.aggregated_data
2022-11-30 13:04:33,309 INFO org.apache.hadoop.hive.ql.Driver: [repro894918]:
Compiling
command(queryId=hive_20221130130433_de2a8b2d-f993-44e5-8aeb-decba3342a85):
INSERT into table default_tmp_compactor_aggregated_data_1669813472898 select
validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
timeofoccurrence) from default.aggregated_data
2022-11-30 13:04:33,314 ERROR org.apache.hadoop.hive.ql.Driver: [repro894918]:
FAILED: ParseException line 1:277 cannot recognize input near 'row' ','
''timeofoccurrence'' in select expression
org.apache.hadoop.hive.ql.parse.ParseException: line 1:277 cannot recognize
input near 'row' ',' ''timeofoccurrence'' in select expression
at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:128)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:82)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74)
at org.apache.hadoop.hive.ql.Compiler.parse(Compiler.java:173)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:102)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:196)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:615)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:673)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:505)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:494)
at
org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:70)
at
org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:133)
at
org.apache.hadoop.hive.ql.txn.compactor.MajorQueryCompactor.runCompaction(MajorQueryCompactor.java:63)
at
org.apache.hadoop.hive.ql.txn.compactor.Worker.findNextCompactionAndExecute(Worker.java:562)
at
org.apache.hadoop.hive.ql.txn.compactor.Worker.lambda$run$0(Worker.java:113)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750){noformat}
was:
Query-based compaction fails on tables having columns with keywords(i.e. row in
this case) for columns. The compaction fails while running insert into
statement as it does not quote the columns correctly.
Below are the steps to reproduce the issue.
{noformat}
CREATE TABLE aggregated_data(`sessionid` string,`row` int,`timeofoccurrence`
bigint);
insert into table aggregated_data values ("abcd",300,211);
insert into table aggregated_data values ("abcd",300,211);
alter table aggregated_data compact 'MAJOR' and wait;{noformat}
Error -
{noformat}
2022-11-30 13:04:33,309 INFO
org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor: [repro894918]: Running
major compaction via query: INSERT into table
default_tmp_compactor_aggregated_data1_1669813472898 select
validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
timeofoccurrence) from default.aggregated_data
2022-11-30 13:04:33,309 INFO org.apache.hadoop.hive.ql.Driver: [repro894918]:
Compiling
command(queryId=hive_20221130130433_de2a8b2d-f993-44e5-8aeb-decba3342a85):
INSERT into table default_tmp_compactor_aggregated_data1_1669813472898 select
[jira] [Work logged] (HIVE-26793) Create a new configuration to override "no compaction" for tables
[
https://issues.apache.org/jira/browse/HIVE-26793?focusedWorklogId=830557=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-830557
]
ASF GitHub Bot logged work on HIVE-26793:
-
Author: ASF GitHub Bot
Created on: 02/Dec/22 08:08
Start Date: 02/Dec/22 08:08
Worklog Time Spent: 10m
Work Description: kokila-19 commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1037821073
##
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics2.java:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.hadoop.hive.metastore.HMSMetricsListener;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.metrics.Metrics;
+import org.apache.hadoop.hive.metastore.metrics.MetricsConstants;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+public class TestCompactionMetrics2 extends CompactorOnTezTest {
Review Comment:
But I see that test class TestCleanerWithReplication in hive-it-unit extends
CompactorTest from hive-exec
Issue Time Tracking
---
Worklog Id: (was: 830557)
Time Spent: 50m (was: 40m)
> Create a new configuration to override "no compaction" for tables
> -
>
> Key: HIVE-26793
> URL: https://issues.apache.org/jira/browse/HIVE-26793
> Project: Hive
> Issue Type: Improvement
>Reporter: Kokila N
>Assignee: Kokila N
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently a simple user can create a table with
> {color:#6a8759}no_auto_compaction=true{color} table property and create an
> aborted write transaction writing to this table. This way a malicious user
> can prevent cleaning up data for the aborted transaction, creating
> performance degradation.
> This configuration should be allowed to overridden on a database level:
> adding {color:#6a8759}no_auto_compaction=false{color} should override the
> table level setting forcing the initiator to schedule compaction for all
> tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26801) Query based compaction fails on tables having columns with keywords(i.e. row in this case)
[
https://issues.apache.org/jira/browse/HIVE-26801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopinath updated HIVE-26801:
Summary: Query based compaction fails on tables having columns with
keywords(i.e. row in this case) (was: Query based compaction fails on tables
having columns with keywords(i.e. row in this case) for columns)
> Query based compaction fails on tables having columns with keywords(i.e. row
> in this case)
> --
>
> Key: HIVE-26801
> URL: https://issues.apache.org/jira/browse/HIVE-26801
> Project: Hive
> Issue Type: Bug
>Reporter: Gopinath
>Assignee: Gopinath
>Priority: Major
>
> Query-based compaction fails on tables having columns with keywords(i.e. row
> in this case) for columns. The compaction fails while running insert into
> statement as it does not quote the columns correctly.
> Below are the steps to reproduce the issue.
> {noformat}
> CREATE TABLE aggregated_data(`sessionid` string,`row` int,`timeofoccurrence`
> bigint);
> insert into table aggregated_data values ("abcd",300,211);
> insert into table aggregated_data values ("abcd",300,211);
> alter table aggregated_data compact 'MAJOR' and wait;{noformat}
> Error -
> {noformat}
> 2022-11-30 13:04:33,309 INFO
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor: [repro894918]:
> Running major compaction via query: INSERT into table
> default_tmp_compactor_aggregated_data1_1669813472898 select
> validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
> ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
> NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
> timeofoccurrence) from default.aggregated_data
> 2022-11-30 13:04:33,309 INFO org.apache.hadoop.hive.ql.Driver:
> [repro894918]: Compiling
> command(queryId=hive_20221130130433_de2a8b2d-f993-44e5-8aeb-decba3342a85):
> INSERT into table default_tmp_compactor_aggregated_data1_1669813472898 select
> validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
> ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
> NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
> timeofoccurrence) from default.aggregated_data
> 2022-11-30 13:04:33,314 ERROR org.apache.hadoop.hive.ql.Driver:
> [repro894918]: FAILED: ParseException line 1:277 cannot recognize input near
> 'row' ',' ''timeofoccurrence'' in select expression
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:277 cannot recognize
> input near 'row' ',' ''timeofoccurrence'' in select expression
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:128)
> at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:82)
> at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74)
> at org.apache.hadoop.hive.ql.Compiler.parse(Compiler.java:173)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:102)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:196)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:615)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:673)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:505)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:494)
> at
> org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:70)
> at
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:133)
> at
> org.apache.hadoop.hive.ql.txn.compactor.MajorQueryCompactor.runCompaction(MajorQueryCompactor.java:63)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.findNextCompactionAndExecute(Worker.java:562)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.lambda$run$0(Worker.java:113)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26801) Query based compaction fails on tables having columns with keywords(i.e. row in this case) for columns
[
https://issues.apache.org/jira/browse/HIVE-26801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopinath reassigned HIVE-26801:
---
> Query based compaction fails on tables having columns with keywords(i.e. row
> in this case) for columns
> --
>
> Key: HIVE-26801
> URL: https://issues.apache.org/jira/browse/HIVE-26801
> Project: Hive
> Issue Type: Bug
>Reporter: Gopinath
>Assignee: Gopinath
>Priority: Major
>
> Query-based compaction fails on tables having columns with keywords(i.e. row
> in this case) for columns. The compaction fails while running insert into
> statement as it does not quote the columns correctly.
> Below are the steps to reproduce the issue.
> {noformat}
> CREATE TABLE aggregated_data(`sessionid` string,`row` int,`timeofoccurrence`
> bigint);
> insert into table aggregated_data values ("abcd",300,211);
> insert into table aggregated_data values ("abcd",300,211);
> alter table aggregated_data compact 'MAJOR' and wait;{noformat}
> Error -
> {noformat}
> 2022-11-30 13:04:33,309 INFO
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor: [repro894918]:
> Running major compaction via query: INSERT into table
> default_tmp_compactor_aggregated_data1_1669813472898 select
> validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
> ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
> NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
> timeofoccurrence) from default.aggregated_data
> 2022-11-30 13:04:33,309 INFO org.apache.hadoop.hive.ql.Driver:
> [repro894918]: Compiling
> command(queryId=hive_20221130130433_de2a8b2d-f993-44e5-8aeb-decba3342a85):
> INSERT into table default_tmp_compactor_aggregated_data1_1669813472898 select
> validate_acid_sort_order(ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId),
> ROW__ID.writeId, ROW__ID.bucketId, ROW__ID.rowId, ROW__ID.writeId,
> NAMED_STRUCT('sessionid', sessionid, 'row', row, 'timeofoccurrence',
> timeofoccurrence) from default.aggregated_data
> 2022-11-30 13:04:33,314 ERROR org.apache.hadoop.hive.ql.Driver:
> [repro894918]: FAILED: ParseException line 1:277 cannot recognize input near
> 'row' ',' ''timeofoccurrence'' in select expression
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:277 cannot recognize
> input near 'row' ',' ''timeofoccurrence'' in select expression
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:128)
> at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:82)
> at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74)
> at org.apache.hadoop.hive.ql.Compiler.parse(Compiler.java:173)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:102)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:196)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:615)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:673)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:505)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:494)
> at
> org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:70)
> at
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:133)
> at
> org.apache.hadoop.hive.ql.txn.compactor.MajorQueryCompactor.runCompaction(MajorQueryCompactor.java:63)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.findNextCompactionAndExecute(Worker.java:562)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.lambda$run$0(Worker.java:113)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
