[jira] [Work logged] (HIVE-23622) Fix orc_merge9
[ https://issues.apache.org/jira/browse/HIVE-23622?focusedWorklogId=837420=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837420 ] ASF GitHub Bot logged work on HIVE-23622: - Author: ASF GitHub Bot Created on: 06/Jan/23 07:50 Start Date: 06/Jan/23 07:50 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3917: URL: https://github.com/apache/hive/pull/3917#issuecomment-1373296202 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3917) [](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3917=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3917=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3917=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3917=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3917=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837420) Time Spent: 0.5h (was: 20m) > Fix orc_merge9 > --- > > Key: HIVE-23622 > URL: https://issues.apache.org/jira/browse/HIVE-23622 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Krisztian Kasa >Priority: Major > Labels: flaky-test, pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26719) Ability to set number of buckets manually
[ https://issues.apache.org/jira/browse/HIVE-26719?focusedWorklogId=837414=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837414 ] ASF GitHub Bot logged work on HIVE-26719: - Author: ASF GitHub Bot Created on: 06/Jan/23 06:09 Start Date: 06/Jan/23 06:09 Worklog Time Spent: 10m Work Description: kasakrisz merged PR #3759: URL: https://github.com/apache/hive/pull/3759 Issue Time Tracking --- Worklog Id: (was: 837414) Time Spent: 1h 50m (was: 1h 40m) > Ability to set number of buckets manually > - > > Key: HIVE-26719 > URL: https://issues.apache.org/jira/browse/HIVE-26719 > Project: Hive > Issue Type: Sub-task >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Add the ability to set the number of implicit buckets manually through the > alter table compact command -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26882) Allow transactional check of Table parameter before altering the Table
[
https://issues.apache.org/jira/browse/HIVE-26882?focusedWorklogId=837413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837413
]
ASF GitHub Bot logged work on HIVE-26882:
-
Author: ASF GitHub Bot
Created on: 06/Jan/23 06:06
Start Date: 06/Jan/23 06:06
Worklog Time Spent: 10m
Work Description: pvary commented on code in PR #3888:
URL: https://github.com/apache/hive/pull/3888#discussion_r1063146796
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:
##
@@ -187,6 +187,16 @@ public void alterTable(RawStore msdb, Warehouse wh, String
catName, String dbnam
TableName.getQualified(catName, dbname, name) + " doesn't exist");
}
+ String expectedKey = environmentContext != null &&
environmentContext.getProperties() != null ?
+
environmentContext.getProperties().get(hive_metastoreConstants.EXPECTED_PARAMETER_KEY)
: null;
+ String expectedValue = environmentContext != null &&
environmentContext.getProperties() != null ?
+
environmentContext.getProperties().get(hive_metastoreConstants.EXPECTED_PARAMETER_VALUE)
: null;
+ if (expectedKey != null && expectedValue != null
+ && !expectedValue.equals(oldt.getParameters().get(expectedKey)))
{
+throw new MetaException("The table has been modified. The parameter
value for key '" + expectedKey + "' is '"
Review Comment:
This is the relevant code part:
```
msdb.openTransaction(Constants.TX_REPEATABLE_READ)
```
Issue Time Tracking
---
Worklog Id: (was: 837413)
Time Spent: 2.5h (was: 2h 20m)
> Allow transactional check of Table parameter before altering the Table
> --
>
> Key: HIVE-26882
> URL: https://issues.apache.org/jira/browse/HIVE-26882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> We should add the possibility to transactionally check if a Table parameter
> is changed before altering the table in the HMS.
> This would provide an alternative, less error-prone and faster way to commit
> an Iceberg table, as the Iceberg table currently needs to:
> - Create an exclusive lock
> - Get the table metadata to check if the current snapshot is not changed
> - Update the table metadata
> - Release the lock
> After the change these 4 HMS calls could be substituted with a single alter
> table call.
> Also we could avoid cases where the locks are left hanging by failed processes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26882) Allow transactional check of Table parameter before altering the Table
[
https://issues.apache.org/jira/browse/HIVE-26882?focusedWorklogId=837412=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837412
]
ASF GitHub Bot logged work on HIVE-26882:
-
Author: ASF GitHub Bot
Created on: 06/Jan/23 06:03
Start Date: 06/Jan/23 06:03
Worklog Time Spent: 10m
Work Description: pvary commented on code in PR #3888:
URL: https://github.com/apache/hive/pull/3888#discussion_r1063145114
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:
##
@@ -187,6 +187,16 @@ public void alterTable(RawStore msdb, Warehouse wh, String
catName, String dbnam
TableName.getQualified(catName, dbname, name) + " doesn't exist");
}
+ String expectedKey = environmentContext != null &&
environmentContext.getProperties() != null ?
+
environmentContext.getProperties().get(hive_metastoreConstants.EXPECTED_PARAMETER_KEY)
: null;
+ String expectedValue = environmentContext != null &&
environmentContext.getProperties() != null ?
+
environmentContext.getProperties().get(hive_metastoreConstants.EXPECTED_PARAMETER_VALUE)
: null;
+ if (expectedKey != null && expectedValue != null
+ && !expectedValue.equals(oldt.getParameters().get(expectedKey)))
{
+throw new MetaException("The table has been modified. The parameter
value for key '" + expectedKey + "' is '"
Review Comment:
Users does not have to do anything. We set the isolation level per
transaction automatically from the code. Similarly as we do in the TxnHandler
methods where the default isolation level is not enough.
Issue Time Tracking
---
Worklog Id: (was: 837412)
Time Spent: 2h 20m (was: 2h 10m)
> Allow transactional check of Table parameter before altering the Table
> --
>
> Key: HIVE-26882
> URL: https://issues.apache.org/jira/browse/HIVE-26882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> We should add the possibility to transactionally check if a Table parameter
> is changed before altering the table in the HMS.
> This would provide an alternative, less error-prone and faster way to commit
> an Iceberg table, as the Iceberg table currently needs to:
> - Create an exclusive lock
> - Get the table metadata to check if the current snapshot is not changed
> - Update the table metadata
> - Release the lock
> After the change these 4 HMS calls could be substituted with a single alter
> table call.
> Also we could avoid cases where the locks are left hanging by failed processes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26911) Renaming a translated external table with a specified location fails with 'location already exists' exception
[
https://issues.apache.org/jira/browse/HIVE-26911?focusedWorklogId=837410=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837410
]
ASF GitHub Bot logged work on HIVE-26911:
-
Author: ASF GitHub Bot
Created on: 06/Jan/23 05:21
Start Date: 06/Jan/23 05:21
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3920:
URL: https://github.com/apache/hive/pull/3920#issuecomment-1373159357
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3920)
[](https://sonarcloud.io/project/issues?id=apache_hive=3920=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3920=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3920=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3920=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3920=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3920=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3920=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3920=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3920=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3920=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3920=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3920=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3920=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3920=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 837410)
Time Spent: 20m (was: 10m)
> Renaming a translated external table with a specified location fails with
> 'location already exists' exception
> -
>
> Key: HIVE-26911
> URL: https://issues.apache.org/jira/browse/HIVE-26911
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Renaming a translated external table with a specified location fails with
> 'location already exists' exception.
> Below are steps for repro
> {code:java}
> create database tmp;
> use tmp;
> create table b(s string) stored as parquet location
> 'hdfs://localhost:20500/test-warehouse/tmp.db/some_location';
> alter table b rename to bb;
> ERROR: InvalidOperationException: New location for this table hive.tmp.bb
> already exists : hdfs://localhost:20500/test-warehouse/tmp.db/some_location
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26908) Disable Initiator on HMS instance at the same time enable Cleaner thread
[
https://issues.apache.org/jira/browse/HIVE-26908?focusedWorklogId=837406=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837406
]
ASF GitHub Bot logged work on HIVE-26908:
-
Author: ASF GitHub Bot
Created on: 06/Jan/23 05:09
Start Date: 06/Jan/23 05:09
Worklog Time Spent: 10m
Work Description: akshat0395 commented on code in PR #3915:
URL: https://github.com/apache/hive/pull/3915#discussion_r1063109875
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/CompactorTasks.java:
##
@@ -110,6 +113,9 @@ private void logCompactionParameters() {
if (!MetastoreConf.getBoolVar(configuration,
MetastoreConf.ConfVars.COMPACTOR_INITIATOR_ON)) {
HiveMetaStore.LOG.warn("Compactor Initiator is turned Off. Automatic
compaction will not be triggered.");
}
+ HiveMetaStore.LOG.info(
Review Comment:
As we are already printing the of COMPACTOR_CLEANER_ON as INFO log on line
96, do we really need this log?
Can this be similar to the WARN log for initiator from line 113 and give a
warn log if cleaner is turned off and automatic cleaner will be be triggered?
Issue Time Tracking
---
Worklog Id: (was: 837406)
Time Spent: 0.5h (was: 20m)
> Disable Initiator on HMS instance at the same time enable Cleaner thread
>
>
> Key: HIVE-26908
> URL: https://issues.apache.org/jira/browse/HIVE-26908
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In the current implementation, both Initiator and Cleaner are either enabled
> or disabled using the same config
> {noformat}
> hive.compactor.initiator.on{noformat}
> So there is no way to selectively disable initiator and enable cleaner or
> vice versa.
> Introducing another config to handle Cleaner thread alone like
> hive.compactor.cleaner.on
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26911) Renaming a translated external table with a specified location fails with 'location already exists' exception
[
https://issues.apache.org/jira/browse/HIVE-26911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26911:
--
Labels: pull-request-available (was: )
> Renaming a translated external table with a specified location fails with
> 'location already exists' exception
> -
>
> Key: HIVE-26911
> URL: https://issues.apache.org/jira/browse/HIVE-26911
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Renaming a translated external table with a specified location fails with
> 'location already exists' exception.
> Below are steps for repro
> {code:java}
> create database tmp;
> use tmp;
> create table b(s string) stored as parquet location
> 'hdfs://localhost:20500/test-warehouse/tmp.db/some_location';
> alter table b rename to bb;
> ERROR: InvalidOperationException: New location for this table hive.tmp.bb
> already exists : hdfs://localhost:20500/test-warehouse/tmp.db/some_location
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26911) Renaming a translated external table with a specified location fails with 'location already exists' exception
[
https://issues.apache.org/jira/browse/HIVE-26911?focusedWorklogId=837405=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837405
]
ASF GitHub Bot logged work on HIVE-26911:
-
Author: ASF GitHub Bot
Created on: 06/Jan/23 04:36
Start Date: 06/Jan/23 04:36
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera opened a new pull request, #3920:
URL: https://github.com/apache/hive/pull/3920
…ation fails with location already exists
### What changes were proposed in this pull request?
Fixes a issue where Renaming a translated external table with a specified
location fails with location already exists exception.
### Why are the changes needed?
Currently we cannot rename a translated external table
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
unit tests
Issue Time Tracking
---
Worklog Id: (was: 837405)
Remaining Estimate: 0h
Time Spent: 10m
> Renaming a translated external table with a specified location fails with
> 'location already exists' exception
> -
>
> Key: HIVE-26911
> URL: https://issues.apache.org/jira/browse/HIVE-26911
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Renaming a translated external table with a specified location fails with
> 'location already exists' exception.
> Below are steps for repro
> {code:java}
> create database tmp;
> use tmp;
> create table b(s string) stored as parquet location
> 'hdfs://localhost:20500/test-warehouse/tmp.db/some_location';
> alter table b rename to bb;
> ERROR: InvalidOperationException: New location for this table hive.tmp.bb
> already exists : hdfs://localhost:20500/test-warehouse/tmp.db/some_location
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26911) Renaming a translated external table with a specified location fails with 'location already exists' exception
[
https://issues.apache.org/jira/browse/HIVE-26911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala reassigned HIVE-26911:
> Renaming a translated external table with a specified location fails with
> 'location already exists' exception
> -
>
> Key: HIVE-26911
> URL: https://issues.apache.org/jira/browse/HIVE-26911
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
>
> Renaming a translated external table with a specified location fails with
> 'location already exists' exception.
> Below are steps for repro
> {code:java}
> create database tmp;
> use tmp;
> create table b(s string) stored as parquet location
> 'hdfs://localhost:20500/test-warehouse/tmp.db/some_location';
> alter table b rename to bb;
> ERROR: InvalidOperationException: New location for this table hive.tmp.bb
> already exists : hdfs://localhost:20500/test-warehouse/tmp.db/some_location
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26065) incorrect metric for active calls —— alter_partitions and alter_partition
[
https://issues.apache.org/jira/browse/HIVE-26065?focusedWorklogId=837385=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837385
]
ASF GitHub Bot logged work on HIVE-26065:
-
Author: ASF GitHub Bot
Created on: 06/Jan/23 02:14
Start Date: 06/Jan/23 02:14
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3919:
URL: https://github.com/apache/hive/pull/3919#issuecomment-1373051563
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3919)
[](https://sonarcloud.io/project/issues?id=apache_hive=3919=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3919=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3919=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3919=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3919=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3919=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3919=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3919=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3919=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3919=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3919=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3919=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3919=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3919=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 837385)
Time Spent: 2h 10m (was: 2h)
> incorrect metric for active calls —— alter_partitions and alter_partition
> ---
>
> Key: HIVE-26065
> URL: https://issues.apache.org/jira/browse/HIVE-26065
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: DongWei
>Assignee: DongWei
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-26065.1.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In our production environment, we found that the value of the metric
> "active_calls_alter_partition" was negative. This is very confusing.
> After checking the HMSHandler.class, we found that the method
> alter_partitions_with_environment_context called()
> *startTableFunction("alter_partitions", catName, db_name, tbl_name);*
> *...*
> *endFunction("alter_partition", oldParts != null, ex, tbl_name);*
> So the metric "active_calls_alter_partition" will be decremented by one after
> the "alter_partitions" function execution ends, causing the value become a
> negative number
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26065) incorrect metric for active calls —— alter_partitions and alter_partition
[
https://issues.apache.org/jira/browse/HIVE-26065?focusedWorklogId=837375=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837375
]
ASF GitHub Bot logged work on HIVE-26065:
-
Author: ASF GitHub Bot
Created on: 06/Jan/23 01:23
Start Date: 06/Jan/23 01:23
Worklog Time Spent: 10m
Work Description: Tuan opened a new pull request, #3919:
URL: https://github.com/apache/hive/pull/3919
In our production environment, we found that the value of the metric
"active_calls_alter_partition" was negative. This is very confusing.
After checking the HMSHandler.class, we found that the method
alter_partitions_with_environment_context called()
startTableFunction("alter_partitions", catName, db_name, tbl_name);
...
endFunction("alter_partition", oldParts != null, ex, tbl_name);
So the metric "active_calls_alter_partition" will be decremented by one
after the "alter_partitions" function execution ends, causing the value become
a negative number
Issue Time Tracking
---
Worklog Id: (was: 837375)
Time Spent: 2h (was: 1h 50m)
> incorrect metric for active calls —— alter_partitions and alter_partition
> ---
>
> Key: HIVE-26065
> URL: https://issues.apache.org/jira/browse/HIVE-26065
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: DongWei
>Assignee: DongWei
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-26065.1.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> In our production environment, we found that the value of the metric
> "active_calls_alter_partition" was negative. This is very confusing.
> After checking the HMSHandler.class, we found that the method
> alter_partitions_with_environment_context called()
> *startTableFunction("alter_partitions", catName, db_name, tbl_name);*
> *...*
> *endFunction("alter_partition", oldParts != null, ex, tbl_name);*
> So the metric "active_calls_alter_partition" will be decremented by one after
> the "alter_partitions" function execution ends, causing the value become a
> negative number
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26065) incorrect metric for active calls —— alter_partitions and alter_partition
[
https://issues.apache.org/jira/browse/HIVE-26065?focusedWorklogId=837366=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837366
]
ASF GitHub Bot logged work on HIVE-26065:
-
Author: ASF GitHub Bot
Created on: 06/Jan/23 00:20
Start Date: 06/Jan/23 00:20
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #3133:
HIVE-26065: fix incorrect metric active_calls_alter_partitions and
active_calls_alter_partition
URL: https://github.com/apache/hive/pull/3133
Issue Time Tracking
---
Worklog Id: (was: 837366)
Time Spent: 1h 50m (was: 1h 40m)
> incorrect metric for active calls —— alter_partitions and alter_partition
> ---
>
> Key: HIVE-26065
> URL: https://issues.apache.org/jira/browse/HIVE-26065
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: DongWei
>Assignee: DongWei
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-26065.1.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> In our production environment, we found that the value of the metric
> "active_calls_alter_partition" was negative. This is very confusing.
> After checking the HMSHandler.class, we found that the method
> alter_partitions_with_environment_context called()
> *startTableFunction("alter_partitions", catName, db_name, tbl_name);*
> *...*
> *endFunction("alter_partition", oldParts != null, ex, tbl_name);*
> So the metric "active_calls_alter_partition" will be decremented by one after
> the "alter_partitions" function execution ends, causing the value become a
> negative number
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26809) Upgrade ORC to 1.8.1
[ https://issues.apache.org/jira/browse/HIVE-26809?focusedWorklogId=837356=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837356 ] ASF GitHub Bot logged work on HIVE-26809: - Author: ASF GitHub Bot Created on: 05/Jan/23 22:42 Start Date: 05/Jan/23 22:42 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3833: URL: https://github.com/apache/hive/pull/3833#issuecomment-1372885398 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3833) [](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3833=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3833=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3833=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3833=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3833=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837356) Time Spent: 3h 10m (was: 3h) > Upgrade ORC to 1.8.1 > > > Key: HIVE-26809 > URL: https://issues.apache.org/jira/browse/HIVE-26809 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Dmitriy Fingerman >Assignee: Dmitriy Fingerman >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26882) Allow transactional check of Table parameter before altering the Table
[
https://issues.apache.org/jira/browse/HIVE-26882?focusedWorklogId=837355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837355
]
ASF GitHub Bot logged work on HIVE-26882:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 22:41
Start Date: 05/Jan/23 22:41
Worklog Time Spent: 10m
Work Description: szehon-ho commented on code in PR #3888:
URL: https://github.com/apache/hive/pull/3888#discussion_r1062793127
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:
##
@@ -187,6 +187,16 @@ public void alterTable(RawStore msdb, Warehouse wh, String
catName, String dbnam
TableName.getQualified(catName, dbname, name) + " doesn't exist");
}
+ String expectedKey = environmentContext != null &&
environmentContext.getProperties() != null ?
+
environmentContext.getProperties().get(hive_metastoreConstants.EXPECTED_PARAMETER_KEY)
: null;
+ String expectedValue = environmentContext != null &&
environmentContext.getProperties() != null ?
+
environmentContext.getProperties().get(hive_metastoreConstants.EXPECTED_PARAMETER_VALUE)
: null;
+ if (expectedKey != null && expectedValue != null
+ && !expectedValue.equals(oldt.getParameters().get(expectedKey)))
{
+throw new MetaException("The table has been modified. The parameter
value for key '" + expectedKey + "' is '"
Review Comment:
Thanks for checking. Is this a setting user can configure in Hive? If so,
it sounds like we should document it as a pre-requisite for user to enable this
feature
Issue Time Tracking
---
Worklog Id: (was: 837355)
Time Spent: 2h 10m (was: 2h)
> Allow transactional check of Table parameter before altering the Table
> --
>
> Key: HIVE-26882
> URL: https://issues.apache.org/jira/browse/HIVE-26882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> We should add the possibility to transactionally check if a Table parameter
> is changed before altering the table in the HMS.
> This would provide an alternative, less error-prone and faster way to commit
> an Iceberg table, as the Iceberg table currently needs to:
> - Create an exclusive lock
> - Get the table metadata to check if the current snapshot is not changed
> - Update the table metadata
> - Release the lock
> After the change these 4 HMS calls could be substituted with a single alter
> table call.
> Also we could avoid cases where the locks are left hanging by failed processes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26903) Compactor threads should gracefully shutdown
[ https://issues.apache.org/jira/browse/HIVE-26903?focusedWorklogId=837345=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837345 ] ASF GitHub Bot logged work on HIVE-26903: - Author: ASF GitHub Bot Created on: 05/Jan/23 22:03 Start Date: 05/Jan/23 22:03 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3916: URL: https://github.com/apache/hive/pull/3916#issuecomment-1372836245 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3916) [](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3916=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3916=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3916=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3916=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3916=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837345) Time Spent: 0.5h (was: 20m) > Compactor threads should gracefully shutdown > > > Key: HIVE-26903 > URL: https://issues.apache.org/jira/browse/HIVE-26903 > Project: Hive > Issue Type: Improvement >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently the compactor threads are daemon threads, which means the JVM will > not wait for these threads to finish. (see: > [https://github.com/apache/hive/blob/431e7d9e5431a808106d8db81e11aea74f040da5/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java#L81)] > As a result during system shutdown, JVM may close all daemon threads > abruptly (JVM won't wait for a thread to sleep/wait, and no > InterruptedException is thrown), so the threads don't have any chance to > shutdown gracefully. This can lead to inconsistent/corrupted state in the > Metastore or on the File system. > Make the compactor threads user threads, and handle shutdown accordingly. > Make sure interrupts and InterruptedException is handled accordingly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26910) Backport HIVE-19104: Use independent warehouse directories in test metastores.
[
https://issues.apache.org/jira/browse/HIVE-26910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HIVE-26910.
--
Resolution: Duplicate
> Backport HIVE-19104: Use independent warehouse directories in test metastores.
> --
>
> Key: HIVE-26910
> URL: https://issues.apache.org/jira/browse/HIVE-26910
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Labels: hive-3.2.0-must, pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {{TestHS2ImpersonationWithRemoteMS}} fails on branch-3. It makes assertions
> about the state of the warehouse directory, but it doesn't account for a part
> of metastore initialization that updates the warehouse directory to
> parameterize it by port number for test isolation.
> {{MetaStoreTestUtils#startMetaStoreWithRetry}} sets the warehouse directory
> as the new {{metastore.warehouse.dir}} property.
> {{AbstractHiveService#get/setWareHouseDir}} later works with the deprecated
> {{hive.metastore.warehouse.dir}} property. {{MetastoreConf}} will take care
> of resolving requests for the new property to values under the old property,
> but not vice versa.
> On master, HIVE-19104 included an additional line in {{MiniHs2}} to make sure
> these 2 properties would stay in sync for test runs. This issue tracks a
> slightly modified backport of that patch to branch-3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26910) Backport HIVE-19104: Use independent warehouse directories in test metastores.
[
https://issues.apache.org/jira/browse/HIVE-26910?focusedWorklogId=837339=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837339
]
ASF GitHub Bot logged work on HIVE-26910:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 21:55
Start Date: 05/Jan/23 21:55
Worklog Time Spent: 10m
Work Description: cnauroth commented on PR #3918:
URL: https://github.com/apache/hive/pull/3918#issuecomment-1372828318
Closing, as this is being done in #3873 .
Issue Time Tracking
---
Worklog Id: (was: 837339)
Time Spent: 40m (was: 0.5h)
> Backport HIVE-19104: Use independent warehouse directories in test metastores.
> --
>
> Key: HIVE-26910
> URL: https://issues.apache.org/jira/browse/HIVE-26910
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Labels: hive-3.2.0-must, pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {{TestHS2ImpersonationWithRemoteMS}} fails on branch-3. It makes assertions
> about the state of the warehouse directory, but it doesn't account for a part
> of metastore initialization that updates the warehouse directory to
> parameterize it by port number for test isolation.
> {{MetaStoreTestUtils#startMetaStoreWithRetry}} sets the warehouse directory
> as the new {{metastore.warehouse.dir}} property.
> {{AbstractHiveService#get/setWareHouseDir}} later works with the deprecated
> {{hive.metastore.warehouse.dir}} property. {{MetastoreConf}} will take care
> of resolving requests for the new property to values under the old property,
> but not vice versa.
> On master, HIVE-19104 included an additional line in {{MiniHs2}} to make sure
> these 2 properties would stay in sync for test runs. This issue tracks a
> slightly modified backport of that patch to branch-3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26910) Backport HIVE-19104: Use independent warehouse directories in test metastores.
[
https://issues.apache.org/jira/browse/HIVE-26910?focusedWorklogId=837340=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837340
]
ASF GitHub Bot logged work on HIVE-26910:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 21:55
Start Date: 05/Jan/23 21:55
Worklog Time Spent: 10m
Work Description: cnauroth closed pull request #3918: HIVE-26910:
Backport HIVE-19104: Use independent warehouse directories in test metastores.
URL: https://github.com/apache/hive/pull/3918
Issue Time Tracking
---
Worklog Id: (was: 837340)
Time Spent: 50m (was: 40m)
> Backport HIVE-19104: Use independent warehouse directories in test metastores.
> --
>
> Key: HIVE-26910
> URL: https://issues.apache.org/jira/browse/HIVE-26910
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Labels: hive-3.2.0-must, pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {{TestHS2ImpersonationWithRemoteMS}} fails on branch-3. It makes assertions
> about the state of the warehouse directory, but it doesn't account for a part
> of metastore initialization that updates the warehouse directory to
> parameterize it by port number for test isolation.
> {{MetaStoreTestUtils#startMetaStoreWithRetry}} sets the warehouse directory
> as the new {{metastore.warehouse.dir}} property.
> {{AbstractHiveService#get/setWareHouseDir}} later works with the deprecated
> {{hive.metastore.warehouse.dir}} property. {{MetastoreConf}} will take care
> of resolving requests for the new property to values under the old property,
> but not vice versa.
> On master, HIVE-19104 included an additional line in {{MiniHs2}} to make sure
> these 2 properties would stay in sync for test runs. This issue tracks a
> slightly modified backport of that patch to branch-3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26869) Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-26869?focusedWorklogId=837337=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837337
]
ASF GitHub Bot logged work on HIVE-26869:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 21:47
Start Date: 05/Jan/23 21:47
Worklog Time Spent: 10m
Work Description: cnauroth commented on PR #3873:
URL: https://github.com/apache/hive/pull/3873#issuecomment-1372818327
> I created [HIVE-26910](https://issues.apache.org/jira/browse/HIVE-26910) /
#3918 for separate tracking of the warehouse directory configuration fix.
Oops, sorry! I misunderstood. I see now that the changes are still included
in this PR. I'll abandon my PR if this one gets committed.
Issue Time Tracking
---
Worklog Id: (was: 837337)
Time Spent: 4h (was: 3h 50m)
> Backport of HIVE-19104: When test MetaStore is started with retry the
> instances should be independent
> -
>
> Key: HIVE-26869
> URL: https://issues.apache.org/jira/browse/HIVE-26869
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
> following error :
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 703.357 s <<< FAILURE! - in
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
> [ERROR]
> testImpersonation(org.apache.hive.service.TestHS2ImpersonationWithRemoteMS)
> Time elapsed: 668.923 s <<< FAILURE!
> java.lang.AssertionError: Unexpected table directory '34015' in warehouse
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation(TestHS2ImpersonationWithRemoteMS.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26910) Backport HIVE-19104: Use independent warehouse directories in test metastores.
[
https://issues.apache.org/jira/browse/HIVE-26910?focusedWorklogId=837338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837338
]
ASF GitHub Bot logged work on HIVE-26910:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 21:47
Start Date: 05/Jan/23 21:47
Worklog Time Spent: 10m
Work Description: cnauroth commented on PR #3918:
URL: https://github.com/apache/hive/pull/3918#issuecomment-1372819245
> @amanraj2520 , FYI on a new branch-3 patch.
@amanraj2520 , Oops, sorry! I misunderstood. I see now that the changes are
still included in your PR. I'll abandon my PR if yours gets committed.
Issue Time Tracking
---
Worklog Id: (was: 837338)
Time Spent: 0.5h (was: 20m)
> Backport HIVE-19104: Use independent warehouse directories in test metastores.
> --
>
> Key: HIVE-26910
> URL: https://issues.apache.org/jira/browse/HIVE-26910
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Labels: hive-3.2.0-must, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {{TestHS2ImpersonationWithRemoteMS}} fails on branch-3. It makes assertions
> about the state of the warehouse directory, but it doesn't account for a part
> of metastore initialization that updates the warehouse directory to
> parameterize it by port number for test isolation.
> {{MetaStoreTestUtils#startMetaStoreWithRetry}} sets the warehouse directory
> as the new {{metastore.warehouse.dir}} property.
> {{AbstractHiveService#get/setWareHouseDir}} later works with the deprecated
> {{hive.metastore.warehouse.dir}} property. {{MetastoreConf}} will take care
> of resolving requests for the new property to values under the old property,
> but not vice versa.
> On master, HIVE-19104 included an additional line in {{MiniHs2}} to make sure
> these 2 properties would stay in sync for test runs. This issue tracks a
> slightly modified backport of that patch to branch-3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26869) Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-26869?focusedWorklogId=837336=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837336
]
ASF GitHub Bot logged work on HIVE-26869:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 21:44
Start Date: 05/Jan/23 21:44
Worklog Time Spent: 10m
Work Description: cnauroth commented on PR #3873:
URL: https://github.com/apache/hive/pull/3873#issuecomment-1372815646
I created [HIVE-26910](https://issues.apache.org/jira/browse/HIVE-26910) /
#3918 for separate tracking of the warehouse directory configuration fix.
Issue Time Tracking
---
Worklog Id: (was: 837336)
Time Spent: 3h 50m (was: 3h 40m)
> Backport of HIVE-19104: When test MetaStore is started with retry the
> instances should be independent
> -
>
> Key: HIVE-26869
> URL: https://issues.apache.org/jira/browse/HIVE-26869
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
> following error :
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 703.357 s <<< FAILURE! - in
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
> [ERROR]
> testImpersonation(org.apache.hive.service.TestHS2ImpersonationWithRemoteMS)
> Time elapsed: 668.923 s <<< FAILURE!
> java.lang.AssertionError: Unexpected table directory '34015' in warehouse
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation(TestHS2ImpersonationWithRemoteMS.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26910) Backport HIVE-19104: Use independent warehouse directories in test metastores.
[
https://issues.apache.org/jira/browse/HIVE-26910?focusedWorklogId=837334=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837334
]
ASF GitHub Bot logged work on HIVE-26910:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 21:43
Start Date: 05/Jan/23 21:43
Worklog Time Spent: 10m
Work Description: cnauroth opened a new pull request, #3918:
URL: https://github.com/apache/hive/pull/3918
### What changes were proposed in this pull request?
`TestHS2ImpersonationWithRemoteMS` fails on branch-3. It makes assertions
about the state of the warehouse directory, but it doesn't account for a part
of metastore initialization that updates the warehouse directory to
parameterize it by port number for test isolation.
`MetaStoreTestUtils#startMetaStoreWithRetry` sets the warehouse directory as
the new `metastore.warehouse.dir` property.
`AbstractHiveService#get/setWareHouseDir` later works with the deprecated
`hive.metastore.warehouse.dir` property. `MetastoreConf` will take care of
resolving requests for the new property to values under the old property, but
not vice versa.
On master, [HIVE-19104](https://issues.apache.org/jira/browse/HIVE-19104)
included an additional line in `MiniHs2` to make sure these 2 properties would
stay in sync for test runs. This is a slightly modified backport of that patch
to branch-3.
There is only one significant difference compared to the master patch. I
needed to resolve a merge conflict in `WarehouseInstance`. This code is
different on branch-3, because it clones a copy of the `HiveConf` before
starting the metastore. (On master, it doesn't make a clone.) I traced this
difference to [HIVE-19812](https://issues.apache.org/jira/browse/HIVE-19812),
in which slightly different patches were committed to master and branch-3:
* master: 9bc90f28cf4406d74f9597837c2b72c809b582e6
* branch-3: 369798d90367187714ba68ff55784cff1342b020
Only the branch-3 patch had a `WarehouseInstance` change. I reviewed
comments on HIVE-19812 / #365, but I couldn't find any record of why branch-3
needed this change. I have decided to preserve the behavior of cloning a copy.
### Why are the changes needed?
This patch is required as part of getting a clean CI run on branch-3 in
preparation for a 3.2.0 release.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Without this patch, `TestHS2ImpersonationWithRemoteMS`fails locally. After
applying the patch, the test passes. I also confirmed successful runs of all
other test suites altered in this patch.
Issue Time Tracking
---
Worklog Id: (was: 837334)
Remaining Estimate: 0h
Time Spent: 10m
> Backport HIVE-19104: Use independent warehouse directories in test metastores.
> --
>
> Key: HIVE-26910
> URL: https://issues.apache.org/jira/browse/HIVE-26910
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Labels: hive-3.2.0-must
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TestHS2ImpersonationWithRemoteMS}} fails on branch-3. It makes assertions
> about the state of the warehouse directory, but it doesn't account for a part
> of metastore initialization that updates the warehouse directory to
> parameterize it by port number for test isolation.
> {{MetaStoreTestUtils#startMetaStoreWithRetry}} sets the warehouse directory
> as the new {{metastore.warehouse.dir}} property.
> {{AbstractHiveService#get/setWareHouseDir}} later works with the deprecated
> {{hive.metastore.warehouse.dir}} property. {{MetastoreConf}} will take care
> of resolving requests for the new property to values under the old property,
> but not vice versa.
> On master, HIVE-19104 included an additional line in {{MiniHs2}} to make sure
> these 2 properties would stay in sync for test runs. This issue tracks a
> slightly modified backport of that patch to branch-3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26910) Backport HIVE-19104: Use independent warehouse directories in test metastores.
[
https://issues.apache.org/jira/browse/HIVE-26910?focusedWorklogId=837335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837335
]
ASF GitHub Bot logged work on HIVE-26910:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 21:43
Start Date: 05/Jan/23 21:43
Worklog Time Spent: 10m
Work Description: cnauroth commented on PR #3918:
URL: https://github.com/apache/hive/pull/3918#issuecomment-1372814418
@amanraj2520 , FYI on a new branch-3 patch.
Issue Time Tracking
---
Worklog Id: (was: 837335)
Time Spent: 20m (was: 10m)
> Backport HIVE-19104: Use independent warehouse directories in test metastores.
> --
>
> Key: HIVE-26910
> URL: https://issues.apache.org/jira/browse/HIVE-26910
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Labels: hive-3.2.0-must, pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{TestHS2ImpersonationWithRemoteMS}} fails on branch-3. It makes assertions
> about the state of the warehouse directory, but it doesn't account for a part
> of metastore initialization that updates the warehouse directory to
> parameterize it by port number for test isolation.
> {{MetaStoreTestUtils#startMetaStoreWithRetry}} sets the warehouse directory
> as the new {{metastore.warehouse.dir}} property.
> {{AbstractHiveService#get/setWareHouseDir}} later works with the deprecated
> {{hive.metastore.warehouse.dir}} property. {{MetastoreConf}} will take care
> of resolving requests for the new property to values under the old property,
> but not vice versa.
> On master, HIVE-19104 included an additional line in {{MiniHs2}} to make sure
> these 2 properties would stay in sync for test runs. This issue tracks a
> slightly modified backport of that patch to branch-3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26910) Backport HIVE-19104: Use independent warehouse directories in test metastores.
[
https://issues.apache.org/jira/browse/HIVE-26910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26910:
--
Labels: hive-3.2.0-must pull-request-available (was: hive-3.2.0-must)
> Backport HIVE-19104: Use independent warehouse directories in test metastores.
> --
>
> Key: HIVE-26910
> URL: https://issues.apache.org/jira/browse/HIVE-26910
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Labels: hive-3.2.0-must, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TestHS2ImpersonationWithRemoteMS}} fails on branch-3. It makes assertions
> about the state of the warehouse directory, but it doesn't account for a part
> of metastore initialization that updates the warehouse directory to
> parameterize it by port number for test isolation.
> {{MetaStoreTestUtils#startMetaStoreWithRetry}} sets the warehouse directory
> as the new {{metastore.warehouse.dir}} property.
> {{AbstractHiveService#get/setWareHouseDir}} later works with the deprecated
> {{hive.metastore.warehouse.dir}} property. {{MetastoreConf}} will take care
> of resolving requests for the new property to values under the old property,
> but not vice versa.
> On master, HIVE-19104 included an additional line in {{MiniHs2}} to make sure
> these 2 properties would stay in sync for test runs. This issue tracks a
> slightly modified backport of that patch to branch-3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26910) Backport HIVE-19104: Use independent warehouse directories in test metastores.
[
https://issues.apache.org/jira/browse/HIVE-26910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HIVE-26910:
-
Parent: HIVE-26836
Issue Type: Sub-task (was: Bug)
> Backport HIVE-19104: Use independent warehouse directories in test metastores.
> --
>
> Key: HIVE-26910
> URL: https://issues.apache.org/jira/browse/HIVE-26910
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Labels: hive-3.2.0-must
>
> {{TestHS2ImpersonationWithRemoteMS}} fails on branch-3. It makes assertions
> about the state of the warehouse directory, but it doesn't account for a part
> of metastore initialization that updates the warehouse directory to
> parameterize it by port number for test isolation.
> {{MetaStoreTestUtils#startMetaStoreWithRetry}} sets the warehouse directory
> as the new {{metastore.warehouse.dir}} property.
> {{AbstractHiveService#get/setWareHouseDir}} later works with the deprecated
> {{hive.metastore.warehouse.dir}} property. {{MetastoreConf}} will take care
> of resolving requests for the new property to values under the old property,
> but not vice versa.
> On master, HIVE-19104 included an additional line in {{MiniHs2}} to make sure
> these 2 properties would stay in sync for test runs. This issue tracks a
> slightly modified backport of that patch to branch-3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26910) Backport HIVE-19104: Use independent warehouse directories in test metastores.
[
https://issues.apache.org/jira/browse/HIVE-26910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth reassigned HIVE-26910:
> Backport HIVE-19104: Use independent warehouse directories in test metastores.
> --
>
> Key: HIVE-26910
> URL: https://issues.apache.org/jira/browse/HIVE-26910
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Labels: hive-3.2.0-must
>
> {{TestHS2ImpersonationWithRemoteMS}} fails on branch-3. It makes assertions
> about the state of the warehouse directory, but it doesn't account for a part
> of metastore initialization that updates the warehouse directory to
> parameterize it by port number for test isolation.
> {{MetaStoreTestUtils#startMetaStoreWithRetry}} sets the warehouse directory
> as the new {{metastore.warehouse.dir}} property.
> {{AbstractHiveService#get/setWareHouseDir}} later works with the deprecated
> {{hive.metastore.warehouse.dir}} property. {{MetastoreConf}} will take care
> of resolving requests for the new property to values under the old property,
> but not vice versa.
> On master, HIVE-19104 included an additional line in {{MiniHs2}} to make sure
> these 2 properties would stay in sync for test runs. This issue tracks a
> slightly modified backport of that patch to branch-3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26802) Create qtest running QB compaction queries
[ https://issues.apache.org/jira/browse/HIVE-26802?focusedWorklogId=837328=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837328 ] ASF GitHub Bot logged work on HIVE-26802: - Author: ASF GitHub Bot Created on: 05/Jan/23 21:08 Start Date: 05/Jan/23 21:08 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3882: URL: https://github.com/apache/hive/pull/3882#issuecomment-1372762360 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3882) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3882=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3882=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3882=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3882=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3882=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837328) Time Spent: 2h 50m (was: 2h 40m) > Create qtest running QB compaction queries > -- > > Key: HIVE-26802 > URL: https://issues.apache.org/jira/browse/HIVE-26802 > Project: Hive > Issue Type: Improvement >Reporter: Zoltán Rátkai >Assignee: Zoltán Rátkai >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > Create a qtest that runs the queries that query-based compaction runs. > Not so much to check for correct data but more to check the query plans, to > simplify tracing changes in compilation that might affect QB compaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26837) CTLT with hive.create.as.external.legacy as true creates managed table instead of external table
[ https://issues.apache.org/jira/browse/HIVE-26837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan resolved HIVE-26837. - Resolution: Fixed > CTLT with hive.create.as.external.legacy as true creates managed table > instead of external table > > > Key: HIVE-26837 > URL: https://issues.apache.org/jira/browse/HIVE-26837 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > When CTLT is used with the config hive.create.as.external.legacy=true, it > still creates managed table by default. Use below to reproduce. > create external table test_ext(empno int, name string) partitioned by(dept > string) stored as orc; > desc formatted test_ext; > set hive.create.as.external.legacy=true; > create table test_external like test_ext; > desc formatted test_external; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-23622) Fix orc_merge9
[ https://issues.apache.org/jira/browse/HIVE-23622?focusedWorklogId=837299=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837299 ] ASF GitHub Bot logged work on HIVE-23622: - Author: ASF GitHub Bot Created on: 05/Jan/23 18:47 Start Date: 05/Jan/23 18:47 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3917: URL: https://github.com/apache/hive/pull/3917#issuecomment-1372601229 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3917) [](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3917=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3917=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3917=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3917=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3917=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3917=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837299) Time Spent: 20m (was: 10m) > Fix orc_merge9 > --- > > Key: HIVE-23622 > URL: https://issues.apache.org/jira/browse/HIVE-23622 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Krisztian Kasa >Priority: Major > Labels: flaky-test, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26837) CTLT with hive.create.as.external.legacy as true creates managed table instead of external table
[ https://issues.apache.org/jira/browse/HIVE-26837?focusedWorklogId=837297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837297 ] ASF GitHub Bot logged work on HIVE-26837: - Author: ASF GitHub Bot Created on: 05/Jan/23 18:33 Start Date: 05/Jan/23 18:33 Worklog Time Spent: 10m Work Description: ramesh0201 merged PR #3854: URL: https://github.com/apache/hive/pull/3854 Issue Time Tracking --- Worklog Id: (was: 837297) Time Spent: 2h 20m (was: 2h 10m) > CTLT with hive.create.as.external.legacy as true creates managed table > instead of external table > > > Key: HIVE-26837 > URL: https://issues.apache.org/jira/browse/HIVE-26837 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > When CTLT is used with the config hive.create.as.external.legacy=true, it > still creates managed table by default. Use below to reproduce. > create external table test_ext(empno int, name string) partitioned by(dept > string) stored as orc; > desc formatted test_ext; > set hive.create.as.external.legacy=true; > create table test_external like test_ext; > desc formatted test_external; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-23622) Fix orc_merge9
[ https://issues.apache.org/jira/browse/HIVE-23622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23622: -- Labels: flaky-test pull-request-available (was: flaky-test) > Fix orc_merge9 > --- > > Key: HIVE-23622 > URL: https://issues.apache.org/jira/browse/HIVE-23622 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Krisztian Kasa >Priority: Major > Labels: flaky-test, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-23622) Fix orc_merge9
[ https://issues.apache.org/jira/browse/HIVE-23622?focusedWorklogId=837294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837294 ] ASF GitHub Bot logged work on HIVE-23622: - Author: ASF GitHub Bot Created on: 05/Jan/23 17:46 Start Date: 05/Jan/23 17:46 Worklog Time Spent: 10m Work Description: kasakrisz opened a new pull request, #3917: URL: https://github.com/apache/hive/pull/3917 ### What changes were proposed in this pull request? Reenable test `orc_merge9.q`. Add the original version of `alltypesorc` file before [HIVE-19399](https://issues.apache.org/jira/browse/HIVE-19339) and use it when loading data in `orc_merge9.q`. ### Why are the changes needed? To test when table has orc files with different metadata those can not be merged. This test depends on the pre generated orc file `hive/data/files/alltypesorc` and this file should have been written by an older `OrcFile.WriterVersion` than the current one used by Hive in order to test the functionality properly. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? ``` mvn test -Dtest.output.overwrite -Dtest=TestMiniLlapLocalCliDriver -Dqfile=orc_merge9.q -pl itests/qtest -Pitests http://ci.hive.apache.org/job/hive-flaky-check/599/ ``` Issue Time Tracking --- Worklog Id: (was: 837294) Remaining Estimate: 0h Time Spent: 10m > Fix orc_merge9 > --- > > Key: HIVE-23622 > URL: https://issues.apache.org/jira/browse/HIVE-23622 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Krisztian Kasa >Priority: Major > Labels: flaky-test > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26809) Upgrade ORC to 1.8.1
[ https://issues.apache.org/jira/browse/HIVE-26809?focusedWorklogId=837293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837293 ] ASF GitHub Bot logged work on HIVE-26809: - Author: ASF GitHub Bot Created on: 05/Jan/23 17:44 Start Date: 05/Jan/23 17:44 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3833: URL: https://github.com/apache/hive/pull/3833#issuecomment-1372533729 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3833) [](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3833=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3833=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3833=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3833=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3833=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3833=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837293) Time Spent: 3h (was: 2h 50m) > Upgrade ORC to 1.8.1 > > > Key: HIVE-26809 > URL: https://issues.apache.org/jira/browse/HIVE-26809 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Dmitriy Fingerman >Assignee: Dmitriy Fingerman >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26875) Transaction conflict retry loop only executes once
[
https://issues.apache.org/jira/browse/HIVE-26875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman resolved HIVE-26875.
-
Resolution: Fixed
Committed to master
> Transaction conflict retry loop only executes once
> --
>
> Key: HIVE-26875
> URL: https://issues.apache.org/jira/browse/HIVE-26875
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently the "conflict retry loop" only executes once.
> [https://github.com/apache/hive/blob/ab4c53de82d4aaa33706510441167f2df55df15e/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L264]
> The intent of this loop is to detect if a conflicting transaction has
> committed while we were waiting to acquire locks. If there is a conflicting
> transaction, it invalidates the snapshot, rolls-back the transaction, opens a
> new transaction and tries to re-acquire locks (and then recompile). It then
> checks again if a conflicting transaction has committed and if so, redoes the
> above steps again, up to HIVE_TXN_MAX_RETRYSNAPSHOT_COUNT times.
> However - isValidTxnState relies on getNonSharedLockedTable():
> [https://github.com/apache/hive/blob/ab4c53de82d4aaa33706510441167f2df55df15e/ql/src/java/org/apache/hadoop/hive/ql/DriverTxnHandler.java#L422]
> which does:
> {code:java}
> private Set getNonSharedLockedTables() {
> if (CollectionUtils.isEmpty(driver.getContext().getHiveLocks())) {
> return Collections.emptySet(); // Nothing to check
> }{code}
> getHiveLocks gets populated by lockAndRespond... HOWEVER -
> compileInternal ends up calling compile which ends up calling
> preparForCompile which ends up calling prepareContext which ends up
> destroying the context with the information lockAndRespond populated. So when
> the loop executes after all of this, it will never detect a 2nd conflict
> because isValidTxnState will always return true (because it thinks there are
> no locked objects).
> This manifests as duplicate records being created during concurrent UPDATEs
> if a transaction get conflicted twice.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26906) Backport of HIVE-19313 to branch-3 : TestJdbcWithDBTokenStoreNoDoAs tests are failing
[
https://issues.apache.org/jira/browse/HIVE-26906?focusedWorklogId=837285=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837285
]
ASF GitHub Bot logged work on HIVE-26906:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 16:26
Start Date: 05/Jan/23 16:26
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3912:
URL: https://github.com/apache/hive/pull/3912#issuecomment-1372440348
@zabetak @abstractdog Can you please review and approve this. A simple
backport.
Issue Time Tracking
---
Worklog Id: (was: 837285)
Time Spent: 0.5h (was: 20m)
> Backport of HIVE-19313 to branch-3 : TestJdbcWithDBTokenStoreNoDoAs tests are
> failing
> -
>
> Key: HIVE-26906
> URL: https://issues.apache.org/jira/browse/HIVE-26906
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> # org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs
> ERROR :
> {{Error Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:42959/default;auth=delegationToken: Peer indicated
> failure: DIGEST-MD5: IO error acquiring password Stacktrace
> java.sql.SQLException: Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:42959/default;auth=delegationToken: Peer indicated
> failure: DIGEST-MD5: IO error acquiring password at
> org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:269) at
> org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) at
> java.sql.DriverManager.getConnection(DriverManager.java:664) at
> java.sql.DriverManager.getConnection(DriverManager.java:270) at
> org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth(TestJdbcWithMiniKdc.java:172)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26899) Backport HIVE-20751 HIVE-23987 Upgrade arrow to 0.11.0 in branch-3
[ https://issues.apache.org/jira/browse/HIVE-26899?focusedWorklogId=837277=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837277 ] ASF GitHub Bot logged work on HIVE-26899: - Author: ASF GitHub Bot Created on: 05/Jan/23 16:07 Start Date: 05/Jan/23 16:07 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #3902: URL: https://github.com/apache/hive/pull/3902#issuecomment-1372416855 @abstractdog Can you please review this. Answered your questions. Issue Time Tracking --- Worklog Id: (was: 837277) Time Spent: 2h (was: 1h 50m) > Backport HIVE-20751 HIVE-23987 Upgrade arrow to 0.11.0 in branch-3 > -- > > Key: HIVE-26899 > URL: https://issues.apache.org/jira/browse/HIVE-26899 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Critical > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26890) Disable TestSSL#testConnectionWrongCertCN (Done as part of HIVE-22621 in master)
[
https://issues.apache.org/jira/browse/HIVE-26890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Raj resolved HIVE-26890.
-
Resolution: Fixed
> Disable TestSSL#testConnectionWrongCertCN (Done as part of HIVE-22621 in
> master)
>
>
> Key: HIVE-26890
> URL: https://issues.apache.org/jira/browse/HIVE-26890
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> TestSSL fails with the following error (this happens in the Hive-3.1.3
> release also, so disabling this test) :
> {code:java}
> [ERROR] Tests run: 10, Failures: 1, Errors: 0, Skipped: 2, Time elapsed:
> 23.143 s <<< FAILURE! - in org.apache.hive.jdbc.TestSSL
> [ERROR] testConnectionWrongCertCN(org.apache.hive.jdbc.TestSSL) Time
> elapsed: 0.64 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN(TestSSL.java:408)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26903) Compactor threads should gracefully shutdown
[ https://issues.apache.org/jira/browse/HIVE-26903?focusedWorklogId=837274=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837274 ] ASF GitHub Bot logged work on HIVE-26903: - Author: ASF GitHub Bot Created on: 05/Jan/23 15:55 Start Date: 05/Jan/23 15:55 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3916: URL: https://github.com/apache/hive/pull/3916#issuecomment-1372399730 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3916) [](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3916=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3916=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3916=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3916=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3916=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3916=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837274) Time Spent: 20m (was: 10m) > Compactor threads should gracefully shutdown > > > Key: HIVE-26903 > URL: https://issues.apache.org/jira/browse/HIVE-26903 > Project: Hive > Issue Type: Improvement >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently the compactor threads are daemon threads, which means the JVM will > not wait for these threads to finish. (see: > [https://github.com/apache/hive/blob/431e7d9e5431a808106d8db81e11aea74f040da5/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java#L81)] > As a result during system shutdown, JVM may close all daemon threads > abruptly (JVM won't wait for a thread to sleep/wait, and no > InterruptedException is thrown), so the threads don't have any chance to > shutdown gracefully. This can lead to inconsistent/corrupted state in the > Metastore or on the File system. > Make the compactor threads user threads, and handle shutdown accordingly. > Make sure interrupts and InterruptedException is handled accordingly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26869) Backport of HIVE-19104: When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-26869?focusedWorklogId=837272=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837272
]
ASF GitHub Bot logged work on HIVE-26869:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 15:51
Start Date: 05/Jan/23 15:51
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3873:
URL: https://github.com/apache/hive/pull/3873#issuecomment-1372395269
@abstractdog Can you please review this PR and merge. We have tested it
locally as well as on the pipelines
Issue Time Tracking
---
Worklog Id: (was: 837272)
Time Spent: 3h 40m (was: 3.5h)
> Backport of HIVE-19104: When test MetaStore is started with retry the
> instances should be independent
> -
>
> Key: HIVE-26869
> URL: https://issues.apache.org/jira/browse/HIVE-26869
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> This fixes TestHS2ImpersonationWithRemoteMS which was failing with the
> following error :
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 703.357 s <<< FAILURE! - in
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
> [ERROR]
> testImpersonation(org.apache.hive.service.TestHS2ImpersonationWithRemoteMS)
> Time elapsed: 668.923 s <<< FAILURE!
> java.lang.AssertionError: Unexpected table directory '34015' in warehouse
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation(TestHS2ImpersonationWithRemoteMS.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26909) Backport of HIVE-20715: Disable test: udaf_histogram_numeric
[ https://issues.apache.org/jira/browse/HIVE-26909?focusedWorklogId=837271=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837271 ] ASF GitHub Bot logged work on HIVE-26909: - Author: ASF GitHub Bot Created on: 05/Jan/23 15:49 Start Date: 05/Jan/23 15:49 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #3914: URL: https://github.com/apache/hive/pull/3914#issuecomment-1372393218 @abstractdog Can you please review and merge this. Issue Time Tracking --- Worklog Id: (was: 837271) Time Spent: 40m (was: 0.5h) > Backport of HIVE-20715: Disable test: udaf_histogram_numeric > - > > Key: HIVE-26909 > URL: https://issues.apache.org/jira/browse/HIVE-26909 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26890) Disable TestSSL#testConnectionWrongCertCN (Done as part of HIVE-22621 in master)
[
https://issues.apache.org/jira/browse/HIVE-26890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-26890:
Fix Version/s: 3.2.0
> Disable TestSSL#testConnectionWrongCertCN (Done as part of HIVE-22621 in
> master)
>
>
> Key: HIVE-26890
> URL: https://issues.apache.org/jira/browse/HIVE-26890
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> TestSSL fails with the following error (this happens in the Hive-3.1.3
> release also, so disabling this test) :
> {code:java}
> [ERROR] Tests run: 10, Failures: 1, Errors: 0, Skipped: 2, Time elapsed:
> 23.143 s <<< FAILURE! - in org.apache.hive.jdbc.TestSSL
> [ERROR] testConnectionWrongCertCN(org.apache.hive.jdbc.TestSSL) Time
> elapsed: 0.64 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN(TestSSL.java:408)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26890) Disable TestSSL#testConnectionWrongCertCN (Done as part of HIVE-22621 in master)
[
https://issues.apache.org/jira/browse/HIVE-26890?focusedWorklogId=837270=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837270
]
ASF GitHub Bot logged work on HIVE-26890:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 15:48
Start Date: 05/Jan/23 15:48
Worklog Time Spent: 10m
Work Description: abstractdog merged PR #3895:
URL: https://github.com/apache/hive/pull/3895
Issue Time Tracking
---
Worklog Id: (was: 837270)
Time Spent: 1h 10m (was: 1h)
> Disable TestSSL#testConnectionWrongCertCN (Done as part of HIVE-22621 in
> master)
>
>
> Key: HIVE-26890
> URL: https://issues.apache.org/jira/browse/HIVE-26890
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> TestSSL fails with the following error (this happens in the Hive-3.1.3
> release also, so disabling this test) :
> {code:java}
> [ERROR] Tests run: 10, Failures: 1, Errors: 0, Skipped: 2, Time elapsed:
> 23.143 s <<< FAILURE! - in org.apache.hive.jdbc.TestSSL
> [ERROR] testConnectionWrongCertCN(org.apache.hive.jdbc.TestSSL) Time
> elapsed: 0.64 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN(TestSSL.java:408)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26890) Disable TestSSL#testConnectionWrongCertCN (Done as part of HIVE-22621 in master)
[
https://issues.apache.org/jira/browse/HIVE-26890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655014#comment-17655014
]
László Bodor commented on HIVE-26890:
-
merged to branch-3, thanks [~amanraj2520]!
> Disable TestSSL#testConnectionWrongCertCN (Done as part of HIVE-22621 in
> master)
>
>
> Key: HIVE-26890
> URL: https://issues.apache.org/jira/browse/HIVE-26890
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> TestSSL fails with the following error (this happens in the Hive-3.1.3
> release also, so disabling this test) :
> {code:java}
> [ERROR] Tests run: 10, Failures: 1, Errors: 0, Skipped: 2, Time elapsed:
> 23.143 s <<< FAILURE! - in org.apache.hive.jdbc.TestSSL
> [ERROR] testConnectionWrongCertCN(org.apache.hive.jdbc.TestSSL) Time
> elapsed: 0.64 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN(TestSSL.java:408)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26890) Disable TestSSL#testConnectionWrongCertCN (Done as part of HIVE-22621 in master)
[
https://issues.apache.org/jira/browse/HIVE-26890?focusedWorklogId=837269=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837269
]
ASF GitHub Bot logged work on HIVE-26890:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 15:47
Start Date: 05/Jan/23 15:47
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3895:
URL: https://github.com/apache/hive/pull/3895#issuecomment-1372390824
@abstractdog Can we please merge this.
Issue Time Tracking
---
Worklog Id: (was: 837269)
Time Spent: 1h (was: 50m)
> Disable TestSSL#testConnectionWrongCertCN (Done as part of HIVE-22621 in
> master)
>
>
> Key: HIVE-26890
> URL: https://issues.apache.org/jira/browse/HIVE-26890
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> TestSSL fails with the following error (this happens in the Hive-3.1.3
> release also, so disabling this test) :
> {code:java}
> [ERROR] Tests run: 10, Failures: 1, Errors: 0, Skipped: 2, Time elapsed:
> 23.143 s <<< FAILURE! - in org.apache.hive.jdbc.TestSSL
> [ERROR] testConnectionWrongCertCN(org.apache.hive.jdbc.TestSSL) Time
> elapsed: 0.64 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN(TestSSL.java:408)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-21263) DruidStorageHandler should not disable the druid datasource when a hive external table is dropped
[
https://issues.apache.org/jira/browse/HIVE-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seung-Hyun Cheong resolved HIVE-21263.
--
Resolution: Won't Fix
> DruidStorageHandler should not disable the druid datasource when a hive
> external table is dropped
> -
>
> Key: HIVE-21263
> URL: https://issues.apache.org/jira/browse/HIVE-21263
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
>
> DruidStorageHandler should not disable the druid datasource of the hive
> external table.
>
> [DruidStorageHandler.java$L678|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L678]
> {code:java}
> if (deleteData && MetaStoreUtils.isExternalTablePurge(table)) {
> LOG.info("Dropping with purge all the data for data source {}",
> dataSourceName);
> List
> dataSegmentList =
> DruidStorageHandlerUtils.getDataSegmentList(getConnector(),
> getDruidMetadataStorageTablesConfig(),
> dataSourceName);
> if (dataSegmentList.isEmpty()) {
> LOG.info("Nothing to delete for data source {}", dataSourceName);
> return;
> }
> for (DataSegment dataSegment : dataSegmentList) {
> try {
> deleteSegment(dataSegment);
> } catch (SegmentLoadingException e) {
> LOG.error(String.format("Error while deleting segment [%s]",
> dataSegment.getIdentifier()), e);
> }
> }
> }
> if (DruidStorageHandlerUtils.disableDataSource(getConnector(),
> getDruidMetadataStorageTablesConfig(),
> dataSourceName)) {
> LOG.info("Successfully dropped druid data source {}", dataSourceName);
> }
> {code}
>
> If a druid datasource is disabled
> # We can not query to the datasource anymore.
> # It takes too much time to enable again a datasource, if the datasource is
> big.
>
> The data of an external table should be usable, even after the external table
> is dropped from hive.
>
> So, DruidStorageHandler should not disable the druid datasource when a hive
> external table is dropped.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC
[
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seung-Hyun Cheong resolved HIVE-21194.
--
Resolution: Won't Fix
> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: image-2019-02-01-16-31-56-958.png,
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0]
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
> type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
> has version[2019-01-31T01:12:32.289+09:00] > task
> version[2019-01-30T16:58:29.992Z]
> at io.druid.indexing.common.task.KillTask.run(KillTask.java:94)
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
> at
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
> [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
> at
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
> [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
> throw new ISE(
> "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
> unusedSegment.getId(),
> unusedSegment.getVersion(),
> myLock.getVersion()
> );
> }
> {code}
>
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new
> DateTime().toString());
> {code}
>
>
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of
> segment to UTC.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26903) Compactor threads should gracefully shutdown
[ https://issues.apache.org/jira/browse/HIVE-26903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26903: -- Labels: pull-request-available (was: ) > Compactor threads should gracefully shutdown > > > Key: HIVE-26903 > URL: https://issues.apache.org/jira/browse/HIVE-26903 > Project: Hive > Issue Type: Improvement >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently the compactor threads are daemon threads, which means the JVM will > not wait for these threads to finish. (see: > [https://github.com/apache/hive/blob/431e7d9e5431a808106d8db81e11aea74f040da5/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java#L81)] > As a result during system shutdown, JVM may close all daemon threads > abruptly (JVM won't wait for a thread to sleep/wait, and no > InterruptedException is thrown), so the threads don't have any chance to > shutdown gracefully. This can lead to inconsistent/corrupted state in the > Metastore or on the File system. > Make the compactor threads user threads, and handle shutdown accordingly. > Make sure interrupts and InterruptedException is handled accordingly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26903) Compactor threads should gracefully shutdown
[ https://issues.apache.org/jira/browse/HIVE-26903?focusedWorklogId=837253=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837253 ] ASF GitHub Bot logged work on HIVE-26903: - Author: ASF GitHub Bot Created on: 05/Jan/23 15:01 Start Date: 05/Jan/23 15:01 Worklog Time Spent: 10m Work Description: veghlaci05 opened a new pull request, #3916: URL: https://github.com/apache/hive/pull/3916 ### What changes were proposed in this pull request? Compactor threads are no longer daemon threads allowing them to shutdown gracefully, and close/cleanup all related resources ### Why are the changes needed? Daemon threads are terminated abruptly without executing the code in catch/finally blocks. As a result compactor threads may leave the metastore in an inconsistent state. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manually and through unit tests Issue Time Tracking --- Worklog Id: (was: 837253) Remaining Estimate: 0h Time Spent: 10m > Compactor threads should gracefully shutdown > > > Key: HIVE-26903 > URL: https://issues.apache.org/jira/browse/HIVE-26903 > Project: Hive > Issue Type: Improvement >Reporter: László Végh >Assignee: László Végh >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently the compactor threads are daemon threads, which means the JVM will > not wait for these threads to finish. (see: > [https://github.com/apache/hive/blob/431e7d9e5431a808106d8db81e11aea74f040da5/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java#L81)] > As a result during system shutdown, JVM may close all daemon threads > abruptly (JVM won't wait for a thread to sleep/wait, and no > InterruptedException is thrown), so the threads don't have any chance to > shutdown gracefully. This can lead to inconsistent/corrupted state in the > Metastore or on the File system. > Make the compactor threads user threads, and handle shutdown accordingly. > Make sure interrupts and InterruptedException is handled accordingly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26887) Make sure dirPath has the correct permissions
[
https://issues.apache.org/jira/browse/HIVE-26887?focusedWorklogId=837251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837251
]
ASF GitHub Bot logged work on HIVE-26887:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 14:49
Start Date: 05/Jan/23 14:49
Worklog Time Spent: 10m
Work Description: skysiders commented on PR #3894:
URL: https://github.com/apache/hive/pull/3894#issuecomment-1372314783
Hi @cnauroth ,thank for your review!
What I want to talk about here is a programming mode, which is implemented
for special permissions. For example, we should check this permission after
setting the special permission. You are right, the 700 permissions here will
not cause defects with a high probability, but I think a good programming style
should be maintained to ensure the correct granting of file permissions, so
here I first judge the file permissions and check the file permissions Whether
it is correctly granted, if not, then set the permissions for them. However, if
the permissions here are changed to 770 or other permissions that may be
affected by the umask in the future, there will be defects. Therefore, from a
rigorous point of view, I think it is necessary to further check the
permissions here.
I have mentioned similar flaws in
[STORM-3862](https://github.com/apache/storm/pull/3478)
,[TEZ-4412](https://github.com/apache/tez/pull/209),[HBASE-26994](https://github.com/apache/hbase/pull/4391)
Finally, regarding the FileSystem.mkdirs(fs, path, perm) you mentioned, I
agree with your point of view, but what I still want to say is that the idea of
this kind of repair is not to set permissions on files, but out of file
permissions Check, as shown in the code, if the permissions are correct, we
don't need to set permissions on it again.
Issue Time Tracking
---
Worklog Id: (was: 837251)
Time Spent: 0.5h (was: 20m)
> Make sure dirPath has the correct permissions
> -
>
> Key: HIVE-26887
> URL: https://issues.apache.org/jira/browse/HIVE-26887
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhang Dongsheng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In the QueryResultsCache function of class QueryResultsCache, there is the
> following code segment
> {code:java}
> private QueryResultsCache(HiveConf configuration) throws IOException {
> ..
> FileSystem fs = cacheDirPath.getFileSystem(conf);
> FsPermission fsPermission = new FsPermission("700");
> fs.mkdirs(cacheDirPath, fsPermission);
> ..
> }
> {code}
> It can be seen that the function will use the mkdirs to create cacheDirPath,
> and the parameters passed in include the path variable cacheDirPath and a
> permission 700. But we haven't confirmed whether the permission is correctly
> assigned to the file.
> The above question is raised because there are two mkdir functions of hadoop,
> {code:java}
> mkdirs(Path f, FsPermission permission)
> {code}
> and
> {code:java}
> mkdirs(FileSystem fs, Path dir, FsPermission permission)
> {code}
> and the first one is used here. The permissions of this function will be
> affected by the underlying umask. Although 700 here will hardly be affected
> by umask, but I think from a rigorous point of view, we should have one more
> permission check and permission grant here.
> And I find same issue in other three methods here.
> In class Context
> {code:java}
> private Path getScratchDir(String scheme, String authority,
> boolean mkdir, String scratchDir) {
> ..
> FileSystem fs = dirPath.getFileSystem(conf);
> dirPath = new Path(fs.makeQualified(dirPath).toString());
> FsPermission fsPermission = new FsPermission(scratchDirPermission);
> if (!fs.mkdirs(dirPath, fsPermission)) {
> throw new RuntimeException("Cannot make directory: "
> + dirPath.toString());
> ..
> }
> {code}
> In class SessionState
> {code:java}
> static void createPath(HiveConf conf, Path path, String permission, boolean
> isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> ..
> if (!fs.mkdirs(path, fsPermission)) {
> throw new IOException("Failed to create directory " + path + " on fs "
> + fs.getUri());
> }
> ..
> }
> {code}
> and in class TezSessionState
> {code:java}
> private Path createTezDir(String sessionId, String suffix) throws IOException
> {
> ..
> Path tezDir = new Path(hdfsScratchDir, TEZ_DIR);
> FileSystem fs = tezDir.getFileSystem(conf);
> FsPermission fsPermission = new
[jira] [Work logged] (HIVE-26802) Create qtest running QB compaction queries
[ https://issues.apache.org/jira/browse/HIVE-26802?focusedWorklogId=837244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837244 ] ASF GitHub Bot logged work on HIVE-26802: - Author: ASF GitHub Bot Created on: 05/Jan/23 14:17 Start Date: 05/Jan/23 14:17 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3882: URL: https://github.com/apache/hive/pull/3882#issuecomment-1372271410 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3882) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3882=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3882=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3882=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3882=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3882=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837244) Time Spent: 2h 40m (was: 2.5h) > Create qtest running QB compaction queries > -- > > Key: HIVE-26802 > URL: https://issues.apache.org/jira/browse/HIVE-26802 > Project: Hive > Issue Type: Improvement >Reporter: Zoltán Rátkai >Assignee: Zoltán Rátkai >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > Create a qtest that runs the queries that query-based compaction runs. > Not so much to check for correct data but more to check the query plans, to > simplify tracing changes in compilation that might affect QB compaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26719) Ability to set number of buckets manually
[ https://issues.apache.org/jira/browse/HIVE-26719?focusedWorklogId=837236=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837236 ] ASF GitHub Bot logged work on HIVE-26719: - Author: ASF GitHub Bot Created on: 05/Jan/23 13:39 Start Date: 05/Jan/23 13:39 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3759: URL: https://github.com/apache/hive/pull/3759#issuecomment-1372226941 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3759) [](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3759=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3759=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3759=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3759=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3759=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837236) Time Spent: 1h 40m (was: 1.5h) > Ability to set number of buckets manually > - > > Key: HIVE-26719 > URL: https://issues.apache.org/jira/browse/HIVE-26719 > Project: Hive > Issue Type: Sub-task >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > Add the ability to set the number of implicit buckets manually through the > alter table compact command -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-23622) Fix orc_merge9
[ https://issues.apache.org/jira/browse/HIVE-23622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa reassigned HIVE-23622: - Assignee: Krisztian Kasa > Fix orc_merge9 > --- > > Key: HIVE-23622 > URL: https://issues.apache.org/jira/browse/HIVE-23622 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Krisztian Kasa >Priority: Major > Labels: flaky-test > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26908) Disable Initiator on HMS instance at the same time enable Cleaner thread
[
https://issues.apache.org/jira/browse/HIVE-26908?focusedWorklogId=837221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837221
]
ASF GitHub Bot logged work on HIVE-26908:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 12:15
Start Date: 05/Jan/23 12:15
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3915:
URL: https://github.com/apache/hive/pull/3915#issuecomment-1372143588
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3915)
[](https://sonarcloud.io/project/issues?id=apache_hive=3915=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3915=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3915=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3915=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3915=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3915=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3915=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3915=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3915=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3915=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3915=false=CODE_SMELL)
[4 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3915=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3915=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3915=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 837221)
Time Spent: 20m (was: 10m)
> Disable Initiator on HMS instance at the same time enable Cleaner thread
>
>
> Key: HIVE-26908
> URL: https://issues.apache.org/jira/browse/HIVE-26908
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In the current implementation, both Initiator and Cleaner are either enabled
> or disabled using the same config
> {noformat}
> hive.compactor.initiator.on{noformat}
> So there is no way to selectively disable initiator and enable cleaner or
> vice versa.
> Introducing another config to handle Cleaner thread alone like
> hive.compactor.cleaner.on
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26808) Port Iceberg catalog changes
[ https://issues.apache.org/jira/browse/HIVE-26808?focusedWorklogId=837207=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837207 ] ASF GitHub Bot logged work on HIVE-26808: - Author: ASF GitHub Bot Created on: 05/Jan/23 11:02 Start Date: 05/Jan/23 11:02 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3907: URL: https://github.com/apache/hive/pull/3907#issuecomment-1372072805 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3907) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3907=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3907=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3907=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3907=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3907=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837207) Time Spent: 50m (was: 40m) > Port Iceberg catalog changes > > > Key: HIVE-26808 > URL: https://issues.apache.org/jira/browse/HIVE-26808 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Zsolt Miskolczi >Assignee: Zsolt Miskolczi >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > The last round of porting happened in 2022 april, there were a couple of > changes especially in HiveTableOperations worth porting into iceberg-catalog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837205
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 10:50
Start Date: 05/Jan/23 10:50
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1062343941
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -5144,6 +5159,7 @@ private int abortTxns(Connection dbConn, List
txnids, boolean checkHeartbe
if (MetastoreConf.getBoolVar(conf,
MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
}
+ LOG.info("Txns are aborted successfully due to : {}", errorMsg);
Review Comment:
You had mentioned that we should avoid persisting due to the below reason -
https://github.com/apache/hive/pull/3656#issuecomment-1274240038
Issue Time Tracking
---
Worklog Id: (was: 837205)
Time Spent: 6.5h (was: 6h 20m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26719) Ability to set number of buckets manually
[ https://issues.apache.org/jira/browse/HIVE-26719?focusedWorklogId=837204=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837204 ] ASF GitHub Bot logged work on HIVE-26719: - Author: ASF GitHub Bot Created on: 05/Jan/23 10:45 Start Date: 05/Jan/23 10:45 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3759: URL: https://github.com/apache/hive/pull/3759#issuecomment-1372056233 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3759) [](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3759=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3759=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3759=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3759=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3759=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3759=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837204) Time Spent: 1.5h (was: 1h 20m) > Ability to set number of buckets manually > - > > Key: HIVE-26719 > URL: https://issues.apache.org/jira/browse/HIVE-26719 > Project: Hive > Issue Type: Sub-task >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Add the ability to set the number of implicit buckets manually through the > alter table compact command -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837203=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837203
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 10:43
Start Date: 05/Jan/23 10:43
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1062338001
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -5144,6 +5159,7 @@ private int abortTxns(Connection dbConn, List
txnids, boolean checkHeartbe
if (MetastoreConf.getBoolVar(conf,
MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
}
+ LOG.info("Txns are aborted successfully due to : {}", errorMsg);
Review Comment:
Could we populate TXN_META_INFO with aborted reason as well?
see //todo: should make abortTxns() write something into TXNS.TXN_META_INFO
about this
Issue Time Tracking
---
Worklog Id: (was: 837203)
Time Spent: 6h 20m (was: 6h 10m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26908) Disable Initiator on HMS instance at the same time enable Cleaner thread
[
https://issues.apache.org/jira/browse/HIVE-26908?focusedWorklogId=837202=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837202
]
ASF GitHub Bot logged work on HIVE-26908:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 10:42
Start Date: 05/Jan/23 10:42
Worklog Time Spent: 10m
Work Description: tarak271 opened a new pull request, #3915:
URL: https://github.com/apache/hive/pull/3915
…e Cleaner thread
### What changes were proposed in this pull request?
Providing config to selectively enable/disable Initiator/Cleaner threads
### Why are the changes needed?
Current implementation uses single config to enable/disable both
Initiator/Cleaner together
### Does this PR introduce _any_ user-facing change?
Yes, for fresh installation/upgrade, customer needs to explicitly set
hive.compactor.cleaner.on to true on HMS instances
### How was this patch tested?
There is no functionality change introduced, so existing test cases should
not be failing with these changes
Issue Time Tracking
---
Worklog Id: (was: 837202)
Remaining Estimate: 0h
Time Spent: 10m
> Disable Initiator on HMS instance at the same time enable Cleaner thread
>
>
> Key: HIVE-26908
> URL: https://issues.apache.org/jira/browse/HIVE-26908
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In the current implementation, both Initiator and Cleaner are either enabled
> or disabled using the same config
> {noformat}
> hive.compactor.initiator.on{noformat}
> So there is no way to selectively disable initiator and enable cleaner or
> vice versa.
> Introducing another config to handle Cleaner thread alone like
> hive.compactor.cleaner.on
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26908) Disable Initiator on HMS instance at the same time enable Cleaner thread
[
https://issues.apache.org/jira/browse/HIVE-26908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26908:
--
Labels: pull-request-available (was: )
> Disable Initiator on HMS instance at the same time enable Cleaner thread
>
>
> Key: HIVE-26908
> URL: https://issues.apache.org/jira/browse/HIVE-26908
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In the current implementation, both Initiator and Cleaner are either enabled
> or disabled using the same config
> {noformat}
> hive.compactor.initiator.on{noformat}
> So there is no way to selectively disable initiator and enable cleaner or
> vice versa.
> Introducing another config to handle Cleaner thread alone like
> hive.compactor.cleaner.on
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837201=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837201
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 10:41
Start Date: 05/Jan/23 10:41
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1062323607
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -5144,6 +5159,7 @@ private int abortTxns(Connection dbConn, List
txnids, boolean checkHeartbe
if (MetastoreConf.getBoolVar(conf,
MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
}
+ LOG.info("Txns are aborted successfully due to : {}", errorMsg);
Review Comment:
The error message is not informative: something aborted because of ...
Maybe print ids of aborted txns? Make sure we are not doing unnecessary
logging. see
performTimeOuts()
LOG.info("Aborted the following transactions due to timeout: {}",
batchToAbort.toString());
LOG.info("Aborted {} transactions due to timeout", numTxnsAborted);
and
commitTxn()
String msg = "Aborting [" + JavaUtils.txnIdToString(txnid) + "," + commitId
+ "]" + " due to a write conflict on " + resource +
" committed by " + committedTxn + " " +
rs.getString(7) + "/" + rs.getString(8);
Issue Time Tracking
---
Worklog Id: (was: 837201)
Time Spent: 6h 10m (was: 6h)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26894) After using scratchdir for staging final job, CTAS and IOW on ACID tables are failing with wrongFS exception
[
https://issues.apache.org/jira/browse/HIVE-26894?focusedWorklogId=837200=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837200
]
ASF GitHub Bot logged work on HIVE-26894:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 10:30
Start Date: 05/Jan/23 10:30
Worklog Time Spent: 10m
Work Description: sankarh commented on code in PR #3909:
URL: https://github.com/apache/hive/pull/3909#discussion_r1062211047
##
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##
@@ -5263,23 +5263,38 @@ public Void call() throws HiveException {
private static void moveAcidFilesForDelta(String deltaFileType, FileSystem
fs,
Path dst, Set
createdDeltaDirs,
-List newFiles, FileStatus
deltaStat) throws HiveException {
-
+List newFiles, FileStatus
deltaStat, HiveConf conf) throws HiveException {
+String configuredOwner = HiveConf.getVar(conf,
ConfVars.HIVE_LOAD_DATA_OWNER);
Path deltaPath = deltaStat.getPath();
// Create the delta directory. Don't worry if it already exists,
// as that likely means another task got to it first. Then move each of
the buckets.
// it would be more efficient to try to move the delta with it's buckets
but that is
// harder to make race condition proof.
Path deltaDest = new Path(dst, deltaPath.getName());
try {
+ FileSystem destFs = deltaDest.getFileSystem(conf);
if (!createdDeltaDirs.contains(deltaDest)) {
try {
- if(fs.mkdirs(deltaDest)) {
+ // Check if the src and dest filesystems are same or not
+ // if the src and dest filesystems are different, then we need to do
copy, instead of rename
+ if (needToCopy(conf, deltaStat.getPath(), deltaDest, fs, destFs,
configuredOwner, true)) {
Review Comment:
Can you add tests that cover these 2 newly added flows?
##
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##
@@ -5263,23 +5263,38 @@ public Void call() throws HiveException {
private static void moveAcidFilesForDelta(String deltaFileType, FileSystem
fs,
Path dst, Set
createdDeltaDirs,
-List newFiles, FileStatus
deltaStat) throws HiveException {
-
+List newFiles, FileStatus
deltaStat, HiveConf conf) throws HiveException {
+String configuredOwner = HiveConf.getVar(conf,
ConfVars.HIVE_LOAD_DATA_OWNER);
Path deltaPath = deltaStat.getPath();
// Create the delta directory. Don't worry if it already exists,
// as that likely means another task got to it first. Then move each of
the buckets.
// it would be more efficient to try to move the delta with it's buckets
but that is
// harder to make race condition proof.
Path deltaDest = new Path(dst, deltaPath.getName());
try {
+ FileSystem destFs = deltaDest.getFileSystem(conf);
if (!createdDeltaDirs.contains(deltaDest)) {
try {
- if(fs.mkdirs(deltaDest)) {
+ // Check if the src and dest filesystems are same or not
+ // if the src and dest filesystems are different, then we need to do
copy, instead of rename
+ if (needToCopy(conf, deltaStat.getPath(), deltaDest, fs, destFs,
configuredOwner, true)) {
+//copy if across file system or encryption zones.
+LOG.debug("Copying source " + deltaStat.getPath() + " to " +
deltaDest + " because HDFS encryption zones are different.");
Review Comment:
The log msg can also highlight, the copy is across different file systems or
encryption zones.
##
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##
@@ -5294,11 +5309,24 @@ private static void moveAcidFilesForDelta(String
deltaFileType, FileSystem fs,
for (FileStatus bucketStat : bucketStats) {
Path bucketSrc = bucketStat.getPath();
Path bucketDest = new Path(deltaDest, bucketSrc.getName());
+FileSystem bucketDestFs = bucketDest.getFileSystem(conf);
LOG.info("Moving bucket " + bucketSrc.toUri().toString() + " to " +
bucketDest.toUri().toString());
try {
- fs.rename(bucketSrc, bucketDest);
- if (newFiles != null) {
+// Check if the src and dest filesystems are same or not
+// if the src and dest filesystems are different, then we need to
do copy, instead of rename
+if (needToCopy(conf, bucketSrc, bucketDest, fs, bucketDestFs,
configuredOwner, true)) {
+ //copy if across file system or encryption zones.
+ LOG.debug("Copying source " + bucketSrc + " to " + bucketDest +
" because HDFS encryption zones are different.");
+
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837197
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 10:29
Start Date: 05/Jan/23 10:29
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1062323607
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -5144,6 +5159,7 @@ private int abortTxns(Connection dbConn, List
txnids, boolean checkHeartbe
if (MetastoreConf.getBoolVar(conf,
MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
}
+ LOG.info("Txns are aborted successfully due to : {}", errorMsg);
Review Comment:
maybe warn + print ids of aborted txns?
Issue Time Tracking
---
Worklog Id: (was: 837197)
Time Spent: 6h (was: 5h 50m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837195=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837195
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 10:28
Start Date: 05/Jan/23 10:28
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1062323607
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -5144,6 +5159,7 @@ private int abortTxns(Connection dbConn, List
txnids, boolean checkHeartbe
if (MetastoreConf.getBoolVar(conf,
MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
}
+ LOG.info("Txns are aborted successfully due to : {}", errorMsg);
Review Comment:
maybe warn?
Issue Time Tracking
---
Worklog Id: (was: 837195)
Time Spent: 5h 50m (was: 5h 40m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837193=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837193
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 10:27
Start Date: 05/Jan/23 10:27
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1062322501
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -5088,12 +5102,13 @@ private int abortTxns(Connection dbConn, List
txnids,
* @throws SQLException
*/
private int abortTxns(Connection dbConn, List txnids, boolean
checkHeartbeat,
-boolean skipCount, boolean isReplReplayed)
+boolean skipCount, boolean isReplReplayed, TxnErrorMsg
errorMsg)
throws SQLException, MetaException {
Statement stmt = null;
if (txnids.isEmpty()) {
return 0;
}
+LOG.info("Trying to abort txns due to : {}", errorMsg);
Review Comment:
maybe warn?
Issue Time Tracking
---
Worklog Id: (was: 837193)
Time Spent: 5.5h (was: 5h 20m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837194=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837194
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 10:27
Start Date: 05/Jan/23 10:27
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1062322501
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -5088,12 +5102,13 @@ private int abortTxns(Connection dbConn, List
txnids,
* @throws SQLException
*/
private int abortTxns(Connection dbConn, List txnids, boolean
checkHeartbeat,
-boolean skipCount, boolean isReplReplayed)
+boolean skipCount, boolean isReplReplayed, TxnErrorMsg
errorMsg)
throws SQLException, MetaException {
Statement stmt = null;
if (txnids.isEmpty()) {
return 0;
}
+LOG.info("Trying to abort txns due to : {}", errorMsg);
Review Comment:
maybe debug here?
Issue Time Tracking
---
Worklog Id: (was: 837194)
Time Spent: 5h 40m (was: 5.5h)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837192=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837192
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 10:26
Start Date: 05/Jan/23 10:26
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1062322501
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -5088,12 +5102,13 @@ private int abortTxns(Connection dbConn, List
txnids,
* @throws SQLException
*/
private int abortTxns(Connection dbConn, List txnids, boolean
checkHeartbeat,
-boolean skipCount, boolean isReplReplayed)
+boolean skipCount, boolean isReplReplayed, TxnErrorMsg
errorMsg)
throws SQLException, MetaException {
Statement stmt = null;
if (txnids.isEmpty()) {
return 0;
}
+LOG.info("Trying to abort txns due to : {}", errorMsg);
Review Comment:
maybe debug?
Issue Time Tracking
---
Worklog Id: (was: 837192)
Time Spent: 5h 20m (was: 5h 10m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837191
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 10:20
Start Date: 05/Jan/23 10:20
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1062317044
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -1087,6 +1096,10 @@ public void abortTxn(AbortTxnRequest rqst) throws
NoSuchTxnException, MetaExcept
@RetrySemantics.Idempotent
public void abortTxns(AbortTxnsRequest rqst) throws MetaException {
List txnIds = rqst.getTxn_ids();
+TxnErrorMsg txnErrorMsg = TxnErrorMsg.NONE;
Review Comment:
in the above method (abortTxn(AbortTxnRequest rqst)), default value was
ABORT_ROLLBACK and here it's NONE. Is this expected?
Issue Time Tracking
---
Worklog Id: (was: 837191)
Time Spent: 5h 10m (was: 5h)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[
https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837190=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837190
]
ASF GitHub Bot logged work on HIVE-11495:
-
Author: ASF GitHub Bot
Created on: 05/Jan/23 10:18
Start Date: 05/Jan/23 10:18
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3908:
URL: https://github.com/apache/hive/pull/3908#discussion_r1062315483
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -1053,7 +1054,15 @@ public void abortTxn(AbortTxnRequest rqst) throws
NoSuchTxnException, MetaExcept
}
raiseTxnUnexpectedState(status, txnid);
}
-abortTxns(dbConn, Collections.singletonList(txnid), true,
isReplayedReplTxn);
+
+if (isReplayedReplTxn) {
+ errorMsg = TxnErrorMsg.ABORT_REPLAYED_REPL_TXN;
+} else if (isHiveReplTxn) {
+ errorMsg = TxnErrorMsg.ABORT_DEFAULT_REPL_TXN;
+} else {
+ errorMsg = TxnErrorMsg.ABORT_ROLLBACK;
Review Comment:
why don't you set it in the var declaration?
Issue Time Tracking
---
Worklog Id: (was: 837190)
Time Spent: 5h (was: 4h 50m)
> Add aborted reason to transaction information.
> --
>
> Key: HIVE-11495
> URL: https://issues.apache.org/jira/browse/HIVE-11495
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Add logging messages mentioning the reason to abort the transaction.
> Another reason: Commit can fail due to a conflicting write from another txn
> (since HIVE-13395)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[ https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837187=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837187 ] ASF GitHub Bot logged work on HIVE-11495: - Author: ASF GitHub Bot Created on: 05/Jan/23 10:13 Start Date: 05/Jan/23 10:13 Worklog Time Spent: 10m Work Description: deniskuzZ commented on code in PR #3908: URL: https://github.com/apache/hive/pull/3908#discussion_r1062308121 ## standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: ## @@ -3227,6 +3227,15 @@ void commitTxn(CommitTxnRequest rqst) */ void abortTxns(List txnids) throws TException; + /** + * Abort a list of transactions with additional information of + * errorcodes as defined in TxnErrorMsg.java. + * @param txnids List of transaction ids + * @param errorCode Error code specifying the reason for abort. + * @throws TException + */ + void abortTxns(List txnids, long errorCode) throws TException; Review Comment: we should make this method generic and use Request object here as well, see CompactionResponse compact2(CompactionRequest request) void dropDatabase(DropDatabaseRequest req) etc Issue Time Tracking --- Worklog Id: (was: 837187) Time Spent: 4h 40m (was: 4.5h) > Add aborted reason to transaction information. > -- > > Key: HIVE-11495 > URL: https://issues.apache.org/jira/browse/HIVE-11495 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Time Spent: 4h 40m > Remaining Estimate: 0h > > Add logging messages mentioning the reason to abort the transaction. > Another reason: Commit can fail due to a conflicting write from another txn > (since HIVE-13395) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[ https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837188=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837188 ] ASF GitHub Bot logged work on HIVE-11495: - Author: ASF GitHub Bot Created on: 05/Jan/23 10:13 Start Date: 05/Jan/23 10:13 Worklog Time Spent: 10m Work Description: deniskuzZ commented on code in PR #3908: URL: https://github.com/apache/hive/pull/3908#discussion_r1062308121 ## standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: ## @@ -3227,6 +3227,15 @@ void commitTxn(CommitTxnRequest rqst) */ void abortTxns(List txnids) throws TException; + /** + * Abort a list of transactions with additional information of + * errorcodes as defined in TxnErrorMsg.java. + * @param txnids List of transaction ids + * @param errorCode Error code specifying the reason for abort. + * @throws TException + */ + void abortTxns(List txnids, long errorCode) throws TException; Review Comment: we should make this method generic and use Request object here as well, see CompactionResponse compact2(CompactionRequest request) void dropDatabase(DropDatabaseRequest req) etc Issue Time Tracking --- Worklog Id: (was: 837188) Time Spent: 4h 50m (was: 4h 40m) > Add aborted reason to transaction information. > -- > > Key: HIVE-11495 > URL: https://issues.apache.org/jira/browse/HIVE-11495 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Time Spent: 4h 50m > Remaining Estimate: 0h > > Add logging messages mentioning the reason to abort the transaction. > Another reason: Commit can fail due to a conflicting write from another txn > (since HIVE-13395) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-11495) Add aborted reason to transaction information.
[ https://issues.apache.org/jira/browse/HIVE-11495?focusedWorklogId=837185=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837185 ] ASF GitHub Bot logged work on HIVE-11495: - Author: ASF GitHub Bot Created on: 05/Jan/23 10:10 Start Date: 05/Jan/23 10:10 Worklog Time Spent: 10m Work Description: deniskuzZ commented on code in PR #3908: URL: https://github.com/apache/hive/pull/3908#discussion_r1062308121 ## standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: ## @@ -3227,6 +3227,15 @@ void commitTxn(CommitTxnRequest rqst) */ void abortTxns(List txnids) throws TException; + /** + * Abort a list of transactions with additional information of + * errorcodes as defined in TxnErrorMsg.java. + * @param txnids List of transaction ids + * @param errorCode Error code specifying the reason for abort. + * @throws TException + */ + void abortTxns(List txnids, long errorCode) throws TException; Review Comment: we should make this method generic and use Request object here as well Issue Time Tracking --- Worklog Id: (was: 837185) Time Spent: 4.5h (was: 4h 20m) > Add aborted reason to transaction information. > -- > > Key: HIVE-11495 > URL: https://issues.apache.org/jira/browse/HIVE-11495 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > Add logging messages mentioning the reason to abort the transaction. > Another reason: Commit can fail due to a conflicting write from another txn > (since HIVE-13395) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26802) Create qtest running QB compaction queries
[ https://issues.apache.org/jira/browse/HIVE-26802?focusedWorklogId=837180=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-837180 ] ASF GitHub Bot logged work on HIVE-26802: - Author: ASF GitHub Bot Created on: 05/Jan/23 09:31 Start Date: 05/Jan/23 09:31 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3882: URL: https://github.com/apache/hive/pull/3882#issuecomment-1371979330 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3882) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3882=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3882=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3882=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3882=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3882=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 837180) Time Spent: 2.5h (was: 2h 20m) > Create qtest running QB compaction queries > -- > > Key: HIVE-26802 > URL: https://issues.apache.org/jira/browse/HIVE-26802 > Project: Hive > Issue Type: Improvement >Reporter: Zoltán Rátkai >Assignee: Zoltán Rátkai >Priority: Minor > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Create a qtest that runs the queries that query-based compaction runs. > Not so much to check for correct data but more to check the query plans, to > simplify tracing changes in compilation that might affect QB compaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
