[jira] [Work logged] (HIVE-26475) syntax issue of msck description
[ https://issues.apache.org/jira/browse/HIVE-26475?focusedWorklogId=823610=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823610 ] ASF GitHub Bot logged work on HIVE-26475: - Author: ASF GitHub Bot Created on: 05/Nov/22 00:25 Start Date: 05/Nov/22 00:25 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #3522: HIVE-26475: Correct msck comment syntax issue URL: https://github.com/apache/hive/pull/3522 Issue Time Tracking --- Worklog Id: (was: 823610) Time Spent: 1h (was: 50m) > syntax issue of msck description > > > Key: HIVE-26475 > URL: https://issues.apache.org/jira/browse/HIVE-26475 > Project: Hive > Issue Type: Improvement > Components: Documentation >Reporter: YuXiang Zhuang >Assignee: YuXiang Zhuang >Priority: Trivial > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > The msck comment has a language disorder, it should be either or instead of > either on -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26707) Iceberg: Write failing due to Ranger Authorization failure
[
https://issues.apache.org/jira/browse/HIVE-26707?focusedWorklogId=823571=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823571
]
ASF GitHub Bot logged work on HIVE-26707:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 21:46
Start Date: 04/Nov/22 21:46
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3731:
URL: https://github.com/apache/hive/pull/3731#issuecomment-1304285907
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3731)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3731=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3731=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3731=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3731=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3731=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 823571)
Time Spent: 50m (was: 40m)
> Iceberg: Write failing due to Ranger Authorization failure
> --
>
> Key: HIVE-26707
> URL: https://issues.apache.org/jira/browse/HIVE-26707
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {noformat}
> Caused by: java.lang.RuntimeException: MetaException(message:Permission
> denied: user [systest] does not have [RWSTORAGE] privilege on
> [iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
> at
> org.apache.iceberg.relocated.com.google.common.base.Throwables.propagate(Throwables.java:243)
> at
> org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:80)
> at
> org.apache.iceberg.hive.MetastoreUtil.alterTable(MetastoreUtil.java:64)
> at
> org.apache.iceberg.hive.HiveTableOperations.lambda$persistTable$3(HiveTableOperations.java:322)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
> at
> org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
>
[jira] [Work logged] (HIVE-26707) Iceberg: Write failing due to Ranger Authorization failure
[
https://issues.apache.org/jira/browse/HIVE-26707?focusedWorklogId=823531=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823531
]
ASF GitHub Bot logged work on HIVE-26707:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 19:28
Start Date: 04/Nov/22 19:28
Worklog Time Spent: 10m
Work Description: ayushtkn commented on PR #3731:
URL: https://github.com/apache/hive/pull/3731#issuecomment-1304064226
Thanx @cnauroth for the review, have added an additional assertion in the
existing test for URI
Issue Time Tracking
---
Worklog Id: (was: 823531)
Time Spent: 40m (was: 0.5h)
> Iceberg: Write failing due to Ranger Authorization failure
> --
>
> Key: HIVE-26707
> URL: https://issues.apache.org/jira/browse/HIVE-26707
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {noformat}
> Caused by: java.lang.RuntimeException: MetaException(message:Permission
> denied: user [systest] does not have [RWSTORAGE] privilege on
> [iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
> at
> org.apache.iceberg.relocated.com.google.common.base.Throwables.propagate(Throwables.java:243)
> at
> org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:80)
> at
> org.apache.iceberg.hive.MetastoreUtil.alterTable(MetastoreUtil.java:64)
> at
> org.apache.iceberg.hive.HiveTableOperations.lambda$persistTable$3(HiveTableOperations.java:322)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
> at
> org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
> at
> org.apache.iceberg.hive.HiveTableOperations.persistTable(HiveTableOperations.java:321)
> at
> org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:277)
> ... 36 more
> Caused by: MetaException(message:Permission denied: user [systest] does not
> have [RWSTORAGE] privilege on
> [iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result$alter_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result$alter_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result.read(ThriftHiveMetastore.java)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table_req(ThriftHiveMetastore.java:2978)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table_req(ThriftHiveMetastore.java:2965)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table_with_environmentContext(HiveMetaStoreClient.java:482)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26707) Iceberg: Write failing due to Ranger Authorization failure
[
https://issues.apache.org/jira/browse/HIVE-26707?focusedWorklogId=823475=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823475
]
ASF GitHub Bot logged work on HIVE-26707:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 16:59
Start Date: 04/Nov/22 16:59
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3731:
URL: https://github.com/apache/hive/pull/3731#issuecomment-1303884769
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3731)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3731=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3731=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3731=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3731=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3731=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 823475)
Time Spent: 0.5h (was: 20m)
> Iceberg: Write failing due to Ranger Authorization failure
> --
>
> Key: HIVE-26707
> URL: https://issues.apache.org/jira/browse/HIVE-26707
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {noformat}
> Caused by: java.lang.RuntimeException: MetaException(message:Permission
> denied: user [systest] does not have [RWSTORAGE] privilege on
> [iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
> at
> org.apache.iceberg.relocated.com.google.common.base.Throwables.propagate(Throwables.java:243)
> at
> org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:80)
> at
> org.apache.iceberg.hive.MetastoreUtil.alterTable(MetastoreUtil.java:64)
> at
> org.apache.iceberg.hive.HiveTableOperations.lambda$persistTable$3(HiveTableOperations.java:322)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
> at
> org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
>
[jira] [Work logged] (HIVE-26704) Cleaner shouldn't be blocked by global min open txnId
[
https://issues.apache.org/jira/browse/HIVE-26704?focusedWorklogId=823441=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823441
]
ASF GitHub Bot logged work on HIVE-26704:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 15:37
Start Date: 04/Nov/22 15:37
Worklog Time Spent: 10m
Work Description: rkirtir commented on code in PR #3576:
URL: https://github.com/apache/hive/pull/3576#discussion_r1014184230
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
##
@@ -384,29 +384,47 @@ public List findReadyToClean(long
minOpenTxnWaterMark, long rete
* By filtering on minOpenTxnWaterMark, we will only cleanup after
every transaction is committed, that could see
* the uncompacted deltas. This way the cleaner can clean up
everything that was made obsolete by this compaction.
*/
-String whereClause = " WHERE \"CQ_STATE\" = '" + READY_FOR_CLEANING +
"'";
-if (minOpenTxnWaterMark > 0) {
+String whereClause = " WHERE \"CQ_STATE\" = " +
quoteChar(READY_FOR_CLEANING) +
Review Comment:
Is it possible to move this bigger query to TxnQueries Class and use
.append()
Issue Time Tracking
---
Worklog Id: (was: 823441)
Time Spent: 20m (was: 10m)
> Cleaner shouldn't be blocked by global min open txnId
> -
>
> Key: HIVE-26704
> URL: https://issues.apache.org/jira/browse/HIVE-26704
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> *Single transaction blocks cluster-wide Cleaner operations*
> Currently, if there is a single long-running transaction that can prevent the
> Cleaner to clean up any tables. This causes file buildup in tables, which can
> cause performance penalties when listing the directories (note that the
> compaction is not blocked by this, so unnecessary data is not read, but the
> files remain there which causes performance penalty).
> We can reduce the protected files from the open transaction if we have
> query-table correlation data stored in the backend DB, but this change will
> need the current method of recording that detail to be revisited.
> The naive and somewhat backward-compatible approach is to capture the
> minOpenWriteIds per table. It involves a non-mutation operation (as in, there
> is no need for the HMS DB to wait for another user’s operation to record it).
> This does spew data writes into the HMS backend DB, but this is a blind
> insert operation that can be group-committed across many users.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26694) Populate file row position information during vectorized Iceberg reads
[ https://issues.apache.org/jira/browse/HIVE-26694?focusedWorklogId=823437=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823437 ] ASF GitHub Bot logged work on HIVE-26694: - Author: ASF GitHub Bot Created on: 04/Nov/22 15:23 Start Date: 04/Nov/22 15:23 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3732: URL: https://github.com/apache/hive/pull/3732#issuecomment-1303756603 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3732) [](https://sonarcloud.io/project/issues?id=apache_hive=3732=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3732=false=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3732=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3732=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3732=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3732=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3732=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3732=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3732=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3732=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3732=false=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3732=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3732=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3732=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 823437) Time Spent: 20m (was: 10m) > Populate file row position information during vectorized Iceberg reads > -- > > Key: HIVE-26694 > URL: https://issues.apache.org/jira/browse/HIVE-26694 > Project: Hive > Issue Type: Sub-task > Components: Iceberg integration >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently we leverage VectorizedOrcRecordReader and > VectorizedParquetRecordReader during vectorized Iceberg reads. In order to > read V2 tables where delete files with positional deletes are present, we > need to make these record reader implementations provide the row number > information too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26694) Populate file row position information during vectorized Iceberg reads
[ https://issues.apache.org/jira/browse/HIVE-26694?focusedWorklogId=823422=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823422 ] ASF GitHub Bot logged work on HIVE-26694: - Author: ASF GitHub Bot Created on: 04/Nov/22 14:26 Start Date: 04/Nov/22 14:26 Worklog Time Spent: 10m Work Description: szlta opened a new pull request, #3732: URL: https://github.com/apache/hive/pull/3732 Currently we leverage VectorizedOrcRecordReader and VectorizedParquetRecordReader during vectorized Iceberg reads. In order to read V2 tables where delete files with positional deletes are present, we need to make these record reader implementations provide the row number information too. Issue Time Tracking --- Worklog Id: (was: 823422) Remaining Estimate: 0h Time Spent: 10m > Populate file row position information during vectorized Iceberg reads > -- > > Key: HIVE-26694 > URL: https://issues.apache.org/jira/browse/HIVE-26694 > Project: Hive > Issue Type: Sub-task > Components: Iceberg integration >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently we leverage VectorizedOrcRecordReader and > VectorizedParquetRecordReader during vectorized Iceberg reads. In order to > read V2 tables where delete files with positional deletes are present, we > need to make these record reader implementations provide the row number > information too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26694) Populate file row position information during vectorized Iceberg reads
[ https://issues.apache.org/jira/browse/HIVE-26694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26694: -- Labels: pull-request-available (was: ) > Populate file row position information during vectorized Iceberg reads > -- > > Key: HIVE-26694 > URL: https://issues.apache.org/jira/browse/HIVE-26694 > Project: Hive > Issue Type: Sub-task > Components: Iceberg integration >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently we leverage VectorizedOrcRecordReader and > VectorizedParquetRecordReader during vectorized Iceberg reads. In order to > read V2 tables where delete files with positional deletes are present, we > need to make these record reader implementations provide the row number > information too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26666) Filter out compactions by id to minimise expense of db operations
[ https://issues.apache.org/jira/browse/HIVE-2?focusedWorklogId=823408=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823408 ] ASF GitHub Bot logged work on HIVE-2: - Author: ASF GitHub Bot Created on: 04/Nov/22 13:52 Start Date: 04/Nov/22 13:52 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3712: URL: https://github.com/apache/hive/pull/3712#issuecomment-1303576985 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3712) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3712=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3712=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3712=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3712=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3712=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 823408) Time Spent: 1h (was: 50m) > Filter out compactions by id to minimise expense of db operations > -- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: KIRTI RUGE >Assignee: KIRTI RUGE >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > At present we use below operations while filtering out compactions in classes > like > AlterTableCompactOperation > cleaner > Use show compaction filter option provided after > https://issues.apache.org/jira/browse/HIVE-13353 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26707) Iceberg: Write failing due to Ranger Authorization failure
[
https://issues.apache.org/jira/browse/HIVE-26707?focusedWorklogId=823410=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823410
]
ASF GitHub Bot logged work on HIVE-26707:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 13:54
Start Date: 04/Nov/22 13:54
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3731:
URL: https://github.com/apache/hive/pull/3731#issuecomment-1303582071
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3731)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3731=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3731=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3731=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3731=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3731=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3731=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 823410)
Time Spent: 20m (was: 10m)
> Iceberg: Write failing due to Ranger Authorization failure
> --
>
> Key: HIVE-26707
> URL: https://issues.apache.org/jira/browse/HIVE-26707
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {noformat}
> Caused by: java.lang.RuntimeException: MetaException(message:Permission
> denied: user [systest] does not have [RWSTORAGE] privilege on
> [iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
> at
> org.apache.iceberg.relocated.com.google.common.base.Throwables.propagate(Throwables.java:243)
> at
> org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:80)
> at
> org.apache.iceberg.hive.MetastoreUtil.alterTable(MetastoreUtil.java:64)
> at
> org.apache.iceberg.hive.HiveTableOperations.lambda$persistTable$3(HiveTableOperations.java:322)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
> at
> org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
>
[jira] [Work logged] (HIVE-26707) Iceberg: Write failing due to Ranger Authorization failure
[
https://issues.apache.org/jira/browse/HIVE-26707?focusedWorklogId=823394=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823394
]
ASF GitHub Bot logged work on HIVE-26707:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 12:58
Start Date: 04/Nov/22 12:58
Worklog Time Spent: 10m
Work Description: ayushtkn opened a new pull request, #3731:
URL: https://github.com/apache/hive/pull/3731
### What changes were proposed in this pull request?
Following the same semantics as Hive-Hbase Handler, encode only the relevant
components rather than the complete URI
### How was this patch tested?
Tested in actual Env
Issue Time Tracking
---
Worklog Id: (was: 823394)
Remaining Estimate: 0h
Time Spent: 10m
> Iceberg: Write failing due to Ranger Authorization failure
> --
>
> Key: HIVE-26707
> URL: https://issues.apache.org/jira/browse/HIVE-26707
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {noformat}
> Caused by: java.lang.RuntimeException: MetaException(message:Permission
> denied: user [systest] does not have [RWSTORAGE] privilege on
> [iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
> at
> org.apache.iceberg.relocated.com.google.common.base.Throwables.propagate(Throwables.java:243)
> at
> org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:80)
> at
> org.apache.iceberg.hive.MetastoreUtil.alterTable(MetastoreUtil.java:64)
> at
> org.apache.iceberg.hive.HiveTableOperations.lambda$persistTable$3(HiveTableOperations.java:322)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
> at
> org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
> at
> org.apache.iceberg.hive.HiveTableOperations.persistTable(HiveTableOperations.java:321)
> at
> org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:277)
> ... 36 more
> Caused by: MetaException(message:Permission denied: user [systest] does not
> have [RWSTORAGE] privilege on
> [iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result$alter_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result$alter_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result.read(ThriftHiveMetastore.java)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table_req(ThriftHiveMetastore.java:2978)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table_req(ThriftHiveMetastore.java:2965)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table_with_environmentContext(HiveMetaStoreClient.java:482)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26707) Iceberg: Write failing due to Ranger Authorization failure
[
https://issues.apache.org/jira/browse/HIVE-26707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26707:
--
Labels: pull-request-available (was: )
> Iceberg: Write failing due to Ranger Authorization failure
> --
>
> Key: HIVE-26707
> URL: https://issues.apache.org/jira/browse/HIVE-26707
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {noformat}
> Caused by: java.lang.RuntimeException: MetaException(message:Permission
> denied: user [systest] does not have [RWSTORAGE] privilege on
> [iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
> at
> org.apache.iceberg.relocated.com.google.common.base.Throwables.propagate(Throwables.java:243)
> at
> org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:80)
> at
> org.apache.iceberg.hive.MetastoreUtil.alterTable(MetastoreUtil.java:64)
> at
> org.apache.iceberg.hive.HiveTableOperations.lambda$persistTable$3(HiveTableOperations.java:322)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
> at
> org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
> at
> org.apache.iceberg.hive.HiveTableOperations.persistTable(HiveTableOperations.java:321)
> at
> org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:277)
> ... 36 more
> Caused by: MetaException(message:Permission denied: user [systest] does not
> have [RWSTORAGE] privilege on
> [iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result$alter_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result$alter_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result.read(ThriftHiveMetastore.java)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table_req(ThriftHiveMetastore.java:2978)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table_req(ThriftHiveMetastore.java:2965)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table_with_environmentContext(HiveMetaStoreClient.java:482)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26707) Iceberg: Write failing due to Ranger Authorization failure
[
https://issues.apache.org/jira/browse/HIVE-26707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-26707:
Description:
{noformat}
Caused by: java.lang.RuntimeException: MetaException(message:Permission denied:
user [systest] does not have [RWSTORAGE] privilege on
[iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
at
org.apache.iceberg.relocated.com.google.common.base.Throwables.propagate(Throwables.java:243)
at
org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:80)
at
org.apache.iceberg.hive.MetastoreUtil.alterTable(MetastoreUtil.java:64)
at
org.apache.iceberg.hive.HiveTableOperations.lambda$persistTable$3(HiveTableOperations.java:322)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
at
org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
at
org.apache.iceberg.hive.HiveTableOperations.persistTable(HiveTableOperations.java:321)
at
org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:277)
... 36 more
Caused by: MetaException(message:Permission denied: user [systest] does not
have [RWSTORAGE] privilege on
[iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result$alter_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result$alter_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table_req(ThriftHiveMetastore.java:2978)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table_req(ThriftHiveMetastore.java:2965)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table_with_environmentContext(HiveMetaStoreClient.java:482)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{noformat}
> Iceberg: Write failing due to Ranger Authorization failure
> --
>
> Key: HIVE-26707
> URL: https://issues.apache.org/jira/browse/HIVE-26707
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
>
> {noformat}
> Caused by: java.lang.RuntimeException: MetaException(message:Permission
> denied: user [systest] does not have [RWSTORAGE] privilege on
> [iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
> at
> org.apache.iceberg.relocated.com.google.common.base.Throwables.propagate(Throwables.java:243)
> at
> org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:80)
> at
> org.apache.iceberg.hive.MetastoreUtil.alterTable(MetastoreUtil.java:64)
> at
> org.apache.iceberg.hive.HiveTableOperations.lambda$persistTable$3(HiveTableOperations.java:322)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
> at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
> at
> org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
> at
> org.apache.iceberg.hive.HiveTableOperations.persistTable(HiveTableOperations.java:321)
> at
> org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:277)
> ... 36 more
> Caused by: MetaException(message:Permission denied: user [systest] does not
> have [RWSTORAGE] privilege on
> [iceberg%3A%2F%2Fspark_iceberg%2Fsimple_types_avro%3Fsnapshot%3D%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Fspark_iceberg.db%2Fsimple_types_avro%2Fmetadata%2F0-50dd01bd-15d9-49bd-a970-80d89231d36c.metadata.json])
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result$alter_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_req_result$alter_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
>
[jira] [Assigned] (HIVE-26707) Iceberg: Write failing due to Ranger Authorization failure
[ https://issues.apache.org/jira/browse/HIVE-26707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena reassigned HIVE-26707: --- > Iceberg: Write failing due to Ranger Authorization failure > -- > > Key: HIVE-26707 > URL: https://issues.apache.org/jira/browse/HIVE-26707 > Project: Hive > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26243) Add vectorized implementation of the 'ds_kll_sketch' UDAF
[ https://issues.apache.org/jira/browse/HIVE-26243?focusedWorklogId=823384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823384 ] ASF GitHub Bot logged work on HIVE-26243: - Author: ASF GitHub Bot Created on: 04/Nov/22 12:21 Start Date: 04/Nov/22 12:21 Worklog Time Spent: 10m Work Description: deniskuzZ commented on code in PR #3317: URL: https://github.com/apache/hive/pull/3317#discussion_r1013960895 ## ql/src/gen/vectorization/UDAFTemplates/VectorUDAFComputeKLL.txt: ## @@ -0,0 +1,307 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates; + +import org.apache.hadoop.hive.common.histogram.KllHistogramEstimator; +import org.apache.hadoop.hive.common.histogram.KllHistogramEstimatorFactory; +import org.apache.hadoop.hive.ql.exec.Description; +import org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector; +import org.apache.hadoop.hive.ql.exec.vector.ColumnVector; +#IF COMPLETE +import org.apache.hadoop.hive.ql.exec.vector.; +#ENDIF COMPLETE +import org.apache.hadoop.hive.ql.exec.vector.VectorAggregationBufferRow; +import org.apache.hadoop.hive.ql.exec.vector.VectorAggregationDesc; +import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch; +import org.apache.hadoop.hive.ql.metadata.HiveException; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator; +import org.apache.hadoop.hive.ql.util.JavaDataModel; + +/** + * Generated from template VectorUDAFComputeKLL.txt. Review Comment: I am not familiar with naming conventions here, but I think Zoli is saying that a more appropriate name would be VectorUDAFComputeDsKllSketch Issue Time Tracking --- Worklog Id: (was: 823384) Time Spent: 3h 20m (was: 3h 10m) > Add vectorized implementation of the 'ds_kll_sketch' UDAF > - > > Key: HIVE-26243 > URL: https://issues.apache.org/jira/browse/HIVE-26243 > Project: Hive > Issue Type: Improvement > Components: UDF, Vectorization >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > _ds_kll_sketch_ UDAF does not have a vectorized implementation at the moment, > the present ticket aims at bridging this gap. > This is particularly important because vectorization has an "all or nothing" > approach, so if this function is used at the side of vectorized functions, > they won't be able to benefit from vectorized execution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=823383=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823383
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 12:15
Start Date: 04/Nov/22 12:15
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1013956124
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7990,19 +7998,29 @@ protected boolean enableColumnStatsCollecting() {
return true;
}
- private Path getCtasLocation(CreateTableDesc tblDesc, boolean
createTableWithSuffix) throws SemanticException {
+ private Path getCtasOrCMVLocation(CreateTableDesc tblDesc,
CreateMaterializedViewDesc viewDesc,
+ boolean createTableWithSuffix) throws
SemanticException {
Path location;
+String protoName;
+String[] names;
+Table tbl;
try {
- String protoName = tblDesc.getDbTableName();
- String[] names = Utilities.getDbTableName(protoName);
-
- // Handle table translation initially and if not present
- // use default table path.
- // Property modifications of the table is handled later.
- // We are interested in the location if it has changed
- // due to table translation.
- Table tbl = tblDesc.toTable(conf);
- tbl = db.getTranslateTableDryrun(tbl.getTTable());
+ if (tblDesc != null) {
+protoName = tblDesc.getDbTableName();
+names = Utilities.getDbTableName(protoName);
Review Comment:
this part is the same for both
Issue Time Tracking
---
Worklog Id: (was: 823383)
Time Spent: 2h (was: 1h 50m)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=823382=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823382
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 12:13
Start Date: 04/Nov/22 12:13
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1013940859
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7610,30 +7610,44 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
destTableIsMaterialization = false;
tableName =
HiveTableName.ofNullableWithNoDefault(viewDesc.getViewName());
tblProps = viewDesc.getTblProps();
+// Add suffix only when required confs are present
+// and user has not specified a location to the table.
+createTableUseSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+&& viewDesc.getLocation() == null;
}
destTableIsTransactional = tblProps != null &&
AcidUtils.isTablePropertyTransactional(tblProps);
if (destTableIsTransactional) {
isNonNativeTable = MetaStoreUtils.isNonNativeTable(tblProps);
boolean isCtas = tblDesc != null && tblDesc.isCTAS();
+boolean isCMV = viewDesc != null && qb.isMaterializedView();
isMmTable = isMmCreate = AcidUtils.isInsertOnlyTable(tblProps);
-if (!isNonNativeTable && !destTableIsTemporary && isCtas) {
+if (!isNonNativeTable && !destTableIsTemporary && (isCtas || isCMV)) {
destTableIsFullAcid = AcidUtils.isFullAcidTable(tblProps);
acidOperation = getAcidType(dest);
- isDirectInsert = isDirectInsert(destTableIsFullAcid, acidOperation);
+ isDirectInsert = isDirectInsertCreate =
isDirectInsert(destTableIsFullAcid, acidOperation);
Review Comment:
minor: what if we have `isCtasOrCMV` and `isDirectInsert`? I think that
would be more readable. by this would be easier to check `isDirectInsert`
related code
Issue Time Tracking
---
Worklog Id: (was: 823382)
Time Spent: 1h 50m (was: 1h 40m)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=823381=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823381
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 12:12
Start Date: 04/Nov/22 12:12
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1013940859
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7610,30 +7610,44 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
destTableIsMaterialization = false;
tableName =
HiveTableName.ofNullableWithNoDefault(viewDesc.getViewName());
tblProps = viewDesc.getTblProps();
+// Add suffix only when required confs are present
+// and user has not specified a location to the table.
+createTableUseSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+&& viewDesc.getLocation() == null;
}
destTableIsTransactional = tblProps != null &&
AcidUtils.isTablePropertyTransactional(tblProps);
if (destTableIsTransactional) {
isNonNativeTable = MetaStoreUtils.isNonNativeTable(tblProps);
boolean isCtas = tblDesc != null && tblDesc.isCTAS();
+boolean isCMV = viewDesc != null && qb.isMaterializedView();
isMmTable = isMmCreate = AcidUtils.isInsertOnlyTable(tblProps);
-if (!isNonNativeTable && !destTableIsTemporary && isCtas) {
+if (!isNonNativeTable && !destTableIsTemporary && (isCtas || isCMV)) {
destTableIsFullAcid = AcidUtils.isFullAcidTable(tblProps);
acidOperation = getAcidType(dest);
- isDirectInsert = isDirectInsert(destTableIsFullAcid, acidOperation);
+ isDirectInsert = isDirectInsertCreate =
isDirectInsert(destTableIsFullAcid, acidOperation);
Review Comment:
minor: what if we have `(isCtas || isCMV)` and `isDirectInsert`? I think
that would be more readable. by this would be easier to check `isDirectInsert`
related code
Issue Time Tracking
---
Worklog Id: (was: 823381)
Time Spent: 1h 40m (was: 1.5h)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=823380=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823380
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 12:11
Start Date: 04/Nov/22 12:11
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1013953352
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7917,17 +7929,13 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
destinationPath, currentTableId, destTableIsFullAcid,
destTableIsTemporary,//this was 1/4 acid
destTableIsMaterialization, queryTmpdir, rsCtx, dpCtx, lbCtx, fsRS,
canBeMerged, destinationTable, writeId, isMmCreate, destType, qb,
isDirectInsert, acidOperation, moveTaskId);
-if (isMmCreate) {
+if (isMmCreate || isDirectInsertCreate) {
Review Comment:
isMmCreate || (isCtas || isCMV) && isDirectInsert?
Issue Time Tracking
---
Worklog Id: (was: 823380)
Time Spent: 1.5h (was: 1h 20m)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=823378=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823378
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 12:04
Start Date: 04/Nov/22 12:04
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1013947907
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7610,30 +7610,44 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
destTableIsMaterialization = false;
tableName =
HiveTableName.ofNullableWithNoDefault(viewDesc.getViewName());
tblProps = viewDesc.getTblProps();
+// Add suffix only when required confs are present
+// and user has not specified a location to the table.
+createTableUseSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+&& viewDesc.getLocation() == null;
}
destTableIsTransactional = tblProps != null &&
AcidUtils.isTablePropertyTransactional(tblProps);
if (destTableIsTransactional) {
isNonNativeTable = MetaStoreUtils.isNonNativeTable(tblProps);
boolean isCtas = tblDesc != null && tblDesc.isCTAS();
+boolean isCMV = viewDesc != null && qb.isMaterializedView();
isMmTable = isMmCreate = AcidUtils.isInsertOnlyTable(tblProps);
-if (!isNonNativeTable && !destTableIsTemporary && isCtas) {
+if (!isNonNativeTable && !destTableIsTemporary && (isCtas || isCMV)) {
destTableIsFullAcid = AcidUtils.isFullAcidTable(tblProps);
acidOperation = getAcidType(dest);
- isDirectInsert = isDirectInsert(destTableIsFullAcid, acidOperation);
+ isDirectInsert = isDirectInsertCreate =
isDirectInsert(destTableIsFullAcid, acidOperation);
if (isDirectInsert || isMmTable) {
-destinationPath = getCtasLocation(tblDesc, createTableUseSuffix);
+destinationPath = getCtasOrCMVLocation(tblDesc, viewDesc,
createTableUseSuffix);
if (createTableUseSuffix) {
- tblDesc.getTblProps().put(SOFT_DELETE_TABLE,
Boolean.TRUE.toString());
+ if (tblDesc != null) {
Review Comment:
maybe isCMV? viewDesc.getTblProps()
Issue Time Tracking
---
Worklog Id: (was: 823378)
Time Spent: 1h 20m (was: 1h 10m)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26669) Hive Metastore become unresponsive
[ https://issues.apache.org/jira/browse/HIVE-26669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Gade updated HIVE-26669: Attachment: (was: metastore-server1) > Hive Metastore become unresponsive > -- > > Key: HIVE-26669 > URL: https://issues.apache.org/jira/browse/HIVE-26669 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.1.0 >Reporter: Sandeep Gade >Assignee: Chris Nauroth >Priority: Critical > > We are experiencing issues with Hive Metastore where it goes unresponsive. > Initial investigation shows thousands of thread in WAITING (parking) state as > shown below: > 1java.lang.Thread.State: BLOCKED (on object monitor) > 772java.lang.Thread.State: RUNNABLE > 2java.lang.Thread.State: TIMED_WAITING (on object monitor) > 13java.lang.Thread.State: TIMED_WAITING (parking) > 5java.lang.Thread.State: TIMED_WAITING (sleeping) > 3java.lang.Thread.State: WAITING (on object monitor) > 14308java.lang.Thread.State: WAITING (parking) > == > Almost all of the threads are stuck at 'parking to wait for > <0x7f9ad0795c48> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)' > > 15 - parking to wait for <0x7f9ad06c9c10> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 14288 - parking to wait for <0x7f9ad0795c48> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > 1 - parking to wait for <0x7f9ad0a161f8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 1 - parking to wait for <0x7f9ad0a39248> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 1 - parking to wait for <0x7f9ad0adb0a0> (a > java.util.concurrent.SynchronousQueue$TransferQueue) > 5 - parking to wait for <0x7f9ad0b12278> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 1 - parking to wait for <0x7f9ad0b12518> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 1 - parking to wait for <0x7f9ad0b44878> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 1 - parking to wait for <0x7f9ad0cbe8f0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 1 - parking to wait for <0x7f9ad1318d60> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 1 - parking to wait for <0x7f9ad1478c10> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 5 - parking to wait for <0x7f9ad1494ff8> (a > java.util.concurrent.SynchronousQueue$TransferQueue) > == > complete stack: > "pool-8-thread-62238" #3582305 prio=5 os_prio=0 tid=0x7f977bfc9800 > nid=0x62011 waiting on condition [0x7f959d917000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7f9ad0795c48> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) > at > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) > at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) > at > org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:351) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:59) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:750) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:718) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:712) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database_core(HiveMetaStore.java:1488) > at >
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=823376=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823376
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 11:59
Start Date: 04/Nov/22 11:59
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1013940859
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7610,30 +7610,44 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
destTableIsMaterialization = false;
tableName =
HiveTableName.ofNullableWithNoDefault(viewDesc.getViewName());
tblProps = viewDesc.getTblProps();
+// Add suffix only when required confs are present
+// and user has not specified a location to the table.
+createTableUseSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+&& viewDesc.getLocation() == null;
}
destTableIsTransactional = tblProps != null &&
AcidUtils.isTablePropertyTransactional(tblProps);
if (destTableIsTransactional) {
isNonNativeTable = MetaStoreUtils.isNonNativeTable(tblProps);
boolean isCtas = tblDesc != null && tblDesc.isCTAS();
+boolean isCMV = viewDesc != null && qb.isMaterializedView();
isMmTable = isMmCreate = AcidUtils.isInsertOnlyTable(tblProps);
-if (!isNonNativeTable && !destTableIsTemporary && isCtas) {
+if (!isNonNativeTable && !destTableIsTemporary && (isCtas || isCMV)) {
destTableIsFullAcid = AcidUtils.isFullAcidTable(tblProps);
acidOperation = getAcidType(dest);
- isDirectInsert = isDirectInsert(destTableIsFullAcid, acidOperation);
+ isDirectInsert = isDirectInsertCreate =
isDirectInsert(destTableIsFullAcid, acidOperation);
Review Comment:
minor: what if we have `isCtasOrCMV` and `isDirectInsert`? I think that
would be more readable. by this would be easier to check `isDirectInsert`
related code
Issue Time Tracking
---
Worklog Id: (was: 823376)
Time Spent: 1h 10m (was: 1h)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26704) Cleaner shouldn't be blocked by global min open txnId
[ https://issues.apache.org/jira/browse/HIVE-26704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko reassigned HIVE-26704: - Assignee: Denys Kuzmenko > Cleaner shouldn't be blocked by global min open txnId > - > > Key: HIVE-26704 > URL: https://issues.apache.org/jira/browse/HIVE-26704 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > *Single transaction blocks cluster-wide Cleaner operations* > Currently, if there is a single long-running transaction that can prevent the > Cleaner to clean up any tables. This causes file buildup in tables, which can > cause performance penalties when listing the directories (note that the > compaction is not blocked by this, so unnecessary data is not read, but the > files remain there which causes performance penalty). > We can reduce the protected files from the open transaction if we have > query-table correlation data stored in the backend DB, but this change will > need the current method of recording that detail to be revisited. > The naive and somewhat backward-compatible approach is to capture the > minOpenWriteIds per table. It involves a non-mutation operation (as in, there > is no need for the HMS DB to wait for another user’s operation to record it). > This does spew data writes into the HMS backend DB, but this is a blind > insert operation that can be group-committed across many users. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26704) Cleaner shouldn't be blocked by global min open txnId
[ https://issues.apache.org/jira/browse/HIVE-26704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26704: -- Description: *Single transaction blocks cluster-wide Cleaner operations* Currently, if there is a single long-running transaction that can prevent the Cleaner to clean up any tables. This causes file buildup in tables, which can cause performance penalties when listing the directories (note that the compaction is not blocked by this, so unnecessary data is not read, but the files remain there which causes performance penalty). We can reduce the protected files from the open transaction if we have query-table correlation data stored in the backend DB, but this change will need the current method of recording that detail to be revisited. The naive and somewhat backward-compatible approach is to capture the minOpenWriteIds per table. It involves a non-mutation operation (as in, there is no need for the HMS DB to wait for another user’s operation to record it). This does spew data writes into the HMS backend DB, but this is a blind insert operation that can be group-committed across many users. > Cleaner shouldn't be blocked by global min open txnId > - > > Key: HIVE-26704 > URL: https://issues.apache.org/jira/browse/HIVE-26704 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > *Single transaction blocks cluster-wide Cleaner operations* > Currently, if there is a single long-running transaction that can prevent the > Cleaner to clean up any tables. This causes file buildup in tables, which can > cause performance penalties when listing the directories (note that the > compaction is not blocked by this, so unnecessary data is not read, but the > files remain there which causes performance penalty). > We can reduce the protected files from the open transaction if we have > query-table correlation data stored in the backend DB, but this change will > need the current method of recording that detail to be revisited. > The naive and somewhat backward-compatible approach is to capture the > minOpenWriteIds per table. It involves a non-mutation operation (as in, there > is no need for the HMS DB to wait for another user’s operation to record it). > This does spew data writes into the HMS backend DB, but this is a blind > insert operation that can be group-committed across many users. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26704) Cleaner shouldn't be blocked by global min open txnId
[ https://issues.apache.org/jira/browse/HIVE-26704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26704: -- Description: *Single transaction blocks cluster-wide Cleaner operations* Currently, if there is a single long-running transaction that can prevent the Cleaner to clean up any tables. This causes file buildup in tables, which can cause performance penalties when listing the directories (note that the compaction is not blocked by this, so unnecessary data is not read, but the files remain there which causes performance penalty). We can reduce the protected files from the open transaction if we have query-table correlation data stored in the backend DB, but this change will need the current method of recording that detail to be revisited. The naive and somewhat backward-compatible approach is to capture the minOpenWriteIds per table. It involves a non-mutation operation (as in, there is no need for the HMS DB to wait for another user’s operation to record it). This does spew data writes into the HMS backend DB, but this is a blind insert operation that can be group-committed across many users. was: *Single transaction blocks cluster-wide Cleaner operations* Currently, if there is a single long-running transaction that can prevent the Cleaner to clean up any tables. This causes file buildup in tables, which can cause performance penalties when listing the directories (note that the compaction is not blocked by this, so unnecessary data is not read, but the files remain there which causes performance penalty). We can reduce the protected files from the open transaction if we have query-table correlation data stored in the backend DB, but this change will need the current method of recording that detail to be revisited. The naive and somewhat backward-compatible approach is to capture the minOpenWriteIds per table. It involves a non-mutation operation (as in, there is no need for the HMS DB to wait for another user’s operation to record it). This does spew data writes into the HMS backend DB, but this is a blind insert operation that can be group-committed across many users. > Cleaner shouldn't be blocked by global min open txnId > - > > Key: HIVE-26704 > URL: https://issues.apache.org/jira/browse/HIVE-26704 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > *Single transaction blocks cluster-wide Cleaner operations* > Currently, if there is a single long-running transaction that can prevent the > Cleaner to clean up any tables. This causes file buildup in tables, which can > cause performance penalties when listing the directories (note that the > compaction is not blocked by this, so unnecessary data is not read, but the > files remain there which causes performance penalty). > We can reduce the protected files from the open transaction if we have > query-table correlation data stored in the backend DB, but this change will > need the current method of recording that detail to be revisited. > The naive and somewhat backward-compatible approach is to capture the > minOpenWriteIds per table. It involves a non-mutation operation (as in, there > is no need for the HMS DB to wait for another user’s operation to record it). > This does spew data writes into the HMS backend DB, but this is a blind > insert operation that can be group-committed across many users. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26706) Add datalake to Hive metadata
[ https://issues.apache.org/jira/browse/HIVE-26706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KIRTI RUGE reassigned HIVE-26706: - Assignee: KIRTI RUGE > Add datalake to Hive metadata > - > > Key: HIVE-26706 > URL: https://issues.apache.org/jira/browse/HIVE-26706 > Project: Hive > Issue Type: New Feature > Components: Database/Schema, Metastore >Reporter: heng.zhao >Assignee: KIRTI RUGE >Priority: Major > > 0: jdbc:hive2://localhost:1> show tables; > +---+---+ > | tab_name | type | > +---+---+ > | test1 | warehouse | > | test2 | iceberg | > +---+---+ > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=823330=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823330
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 09:50
Start Date: 04/Nov/22 09:50
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1013814692
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7610,30 +7610,44 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
destTableIsMaterialization = false;
tableName =
HiveTableName.ofNullableWithNoDefault(viewDesc.getViewName());
tblProps = viewDesc.getTblProps();
+// Add suffix only when required confs are present
+// and user has not specified a location to the table.
+createTableUseSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+&& viewDesc.getLocation() == null;
}
destTableIsTransactional = tblProps != null &&
AcidUtils.isTablePropertyTransactional(tblProps);
if (destTableIsTransactional) {
isNonNativeTable = MetaStoreUtils.isNonNativeTable(tblProps);
boolean isCtas = tblDesc != null && tblDesc.isCTAS();
+boolean isCMV = viewDesc != null && qb.isMaterializedView();
isMmTable = isMmCreate = AcidUtils.isInsertOnlyTable(tblProps);
-if (!isNonNativeTable && !destTableIsTemporary && isCtas) {
+if (!isNonNativeTable && !destTableIsTemporary && (isCtas || isCMV)) {
destTableIsFullAcid = AcidUtils.isFullAcidTable(tblProps);
acidOperation = getAcidType(dest);
- isDirectInsert = isDirectInsert(destTableIsFullAcid, acidOperation);
+ isDirectInsert = isDirectInsertCreate =
isDirectInsert(destTableIsFullAcid, acidOperation);
Review Comment:
Ok, got it, thank you!
Issue Time Tracking
---
Worklog Id: (was: 823330)
Time Spent: 1h (was: 50m)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26666) Filter out compactions by id to minimise expense of db operations
[ https://issues.apache.org/jira/browse/HIVE-2?focusedWorklogId=823319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823319 ] ASF GitHub Bot logged work on HIVE-2: - Author: ASF GitHub Bot Created on: 04/Nov/22 09:28 Start Date: 04/Nov/22 09:28 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3712: URL: https://github.com/apache/hive/pull/3712#issuecomment-1303168129 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3712) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3712=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3712=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3712=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3712=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3712=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 823319) Time Spent: 50m (was: 40m) > Filter out compactions by id to minimise expense of db operations > -- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: KIRTI RUGE >Assignee: KIRTI RUGE >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > At present we use below operations while filtering out compactions in classes > like > AlterTableCompactOperation > cleaner > Use show compaction filter option provided after > https://issues.apache.org/jira/browse/HIVE-13353 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26628) Iceberg table is created when running explain ctas command
[
https://issues.apache.org/jira/browse/HIVE-26628?focusedWorklogId=823307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823307
]
ASF GitHub Bot logged work on HIVE-26628:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 08:27
Start Date: 04/Nov/22 08:27
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3724:
URL: https://github.com/apache/hive/pull/3724#issuecomment-1303111760
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3724)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3724=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3724=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3724=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3724=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3724=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 823307)
Time Spent: 2h 10m (was: 2h)
> Iceberg table is created when running explain ctas command
> --
>
> Key: HIVE-26628
> URL: https://issues.apache.org/jira/browse/HIVE-26628
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> {code}
> create table source(a int, b string, c int);
> explain
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> {code}
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Table already exists:
> default.tbl_ice
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:13963)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12528)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12693)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
> at
>
[jira] [Work logged] (HIVE-26648) Upgrade Bouncy Castle to 1.70 due to high CVEs
[ https://issues.apache.org/jira/browse/HIVE-26648?focusedWorklogId=823295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823295 ] ASF GitHub Bot logged work on HIVE-26648: - Author: ASF GitHub Bot Created on: 04/Nov/22 08:08 Start Date: 04/Nov/22 08:08 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3727: URL: https://github.com/apache/hive/pull/3727#issuecomment-1303096976 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3727) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3727=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3727=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3727=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3727=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3727=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 823295) Time Spent: 1.5h (was: 1h 20m) > Upgrade Bouncy Castle to 1.70 due to high CVEs > --- > > Key: HIVE-26648 > URL: https://issues.apache.org/jira/browse/HIVE-26648 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=823288=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823288
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 07:57
Start Date: 04/Nov/22 07:57
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1013714860
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7610,30 +7610,44 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
destTableIsMaterialization = false;
tableName =
HiveTableName.ofNullableWithNoDefault(viewDesc.getViewName());
tblProps = viewDesc.getTblProps();
+// Add suffix only when required confs are present
+// and user has not specified a location to the table.
+createTableUseSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+&& viewDesc.getLocation() == null;
}
destTableIsTransactional = tblProps != null &&
AcidUtils.isTablePropertyTransactional(tblProps);
if (destTableIsTransactional) {
isNonNativeTable = MetaStoreUtils.isNonNativeTable(tblProps);
boolean isCtas = tblDesc != null && tblDesc.isCTAS();
+boolean isCMV = viewDesc != null && qb.isMaterializedView();
isMmTable = isMmCreate = AcidUtils.isInsertOnlyTable(tblProps);
-if (!isNonNativeTable && !destTableIsTemporary && isCtas) {
+if (!isNonNativeTable && !destTableIsTemporary && (isCtas || isCMV)) {
destTableIsFullAcid = AcidUtils.isFullAcidTable(tblProps);
acidOperation = getAcidType(dest);
- isDirectInsert = isDirectInsert(destTableIsFullAcid, acidOperation);
+ isDirectInsert = isDirectInsertCreate =
isDirectInsert(destTableIsFullAcid, acidOperation);
Review Comment:
Thanks for taking a look at this - @veghlaci05
This variable is required to differentiate between direct insert which is
initiated by insert commands and direct insert initiated by CTAS/CMV.
`isDirectInsertCreate` is by default false and only set when CTAS or CMV is
initiated.
Earlier there used to be a specific check for direct insert with CTAS here -
https://github.com/apache/hive/pull/3715/files#diff-d4b1a32bbbd9e283893a6b52854c7aeb3e356a1ba1add2c4107e52901ca268f9L7927-L7930
Since the idea is to set the writer only when direct insert is initiated by
CTAS or CMV, this can be achieved by using a specific variable to distinguish
between direct insert initiated by Insert commands and direct insert initiated
by CTAS/CMV.
Issue Time Tracking
---
Worklog Id: (was: 823288)
Time Spent: 50m (was: 40m)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26680) Make CMV use Direct Insert Semantics
[
https://issues.apache.org/jira/browse/HIVE-26680?focusedWorklogId=823283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823283
]
ASF GitHub Bot logged work on HIVE-26680:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 07:46
Start Date: 04/Nov/22 07:46
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #3715:
URL: https://github.com/apache/hive/pull/3715#discussion_r1013699818
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7610,30 +7610,44 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
destTableIsMaterialization = false;
tableName =
HiveTableName.ofNullableWithNoDefault(viewDesc.getViewName());
tblProps = viewDesc.getTblProps();
+// Add suffix only when required confs are present
+// and user has not specified a location to the table.
+createTableUseSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+&& viewDesc.getLocation() == null;
}
destTableIsTransactional = tblProps != null &&
AcidUtils.isTablePropertyTransactional(tblProps);
if (destTableIsTransactional) {
isNonNativeTable = MetaStoreUtils.isNonNativeTable(tblProps);
boolean isCtas = tblDesc != null && tblDesc.isCTAS();
+boolean isCMV = viewDesc != null && qb.isMaterializedView();
isMmTable = isMmCreate = AcidUtils.isInsertOnlyTable(tblProps);
-if (!isNonNativeTable && !destTableIsTemporary && isCtas) {
+if (!isNonNativeTable && !destTableIsTemporary && (isCtas || isCMV)) {
destTableIsFullAcid = AcidUtils.isFullAcidTable(tblProps);
acidOperation = getAcidType(dest);
- isDirectInsert = isDirectInsert(destTableIsFullAcid, acidOperation);
+ isDirectInsert = isDirectInsertCreate =
isDirectInsert(destTableIsFullAcid, acidOperation);
Review Comment:
What is the purpose of this new variable? As far as I can see it is always
the same as isDirectInsert. Am I missing sth?
Issue Time Tracking
---
Worklog Id: (was: 823283)
Time Spent: 40m (was: 0.5h)
> Make CMV use Direct Insert Semantics
>
>
> Key: HIVE-26680
> URL: https://issues.apache.org/jira/browse/HIVE-26680
> Project: Hive
> Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> As similar to CTAS, CMV can also use Direct Insert approach which can
> potentially save a lot of filesystem operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26580) SHOW COMPACTIONS should support ordering and limiting functionality in filtering options
[ https://issues.apache.org/jira/browse/HIVE-26580?focusedWorklogId=823277=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823277 ] ASF GitHub Bot logged work on HIVE-26580: - Author: ASF GitHub Bot Created on: 04/Nov/22 07:07 Start Date: 04/Nov/22 07:07 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3708: URL: https://github.com/apache/hive/pull/3708#issuecomment-1303052702 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3708) [](https://sonarcloud.io/project/issues?id=apache_hive=3708=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3708=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3708=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3708=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3708=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3708=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3708=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3708=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3708=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3708=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3708=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=3708=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3708=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3708=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 823277) Time Spent: 2h 40m (was: 2.5h) > SHOW COMPACTIONS should support ordering and limiting functionality in > filtering options > > > Key: HIVE-26580 > URL: https://issues.apache.org/jira/browse/HIVE-26580 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: KIRTI RUGE >Assignee: KIRTI RUGE >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > SHOW COMPACTION should provide ordering by defied table . It should also > support limitation of fetched records -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26666) Filter out compactions by id to minimise expense of db operations
[ https://issues.apache.org/jira/browse/HIVE-2?focusedWorklogId=823275=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823275 ] ASF GitHub Bot logged work on HIVE-2: - Author: ASF GitHub Bot Created on: 04/Nov/22 06:48 Start Date: 04/Nov/22 06:48 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3712: URL: https://github.com/apache/hive/pull/3712#issuecomment-1303041890 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3712) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3712=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3712=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3712=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=3712=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3712=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3712=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 823275) Time Spent: 40m (was: 0.5h) > Filter out compactions by id to minimise expense of db operations > -- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: KIRTI RUGE >Assignee: KIRTI RUGE >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > At present we use below operations while filtering out compactions in classes > like > AlterTableCompactOperation > cleaner > Use show compaction filter option provided after > https://issues.apache.org/jira/browse/HIVE-13353 -- This message was sent by Atlassian Jira (v8.20.10#820010)
