Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]
lirui-apache commented on code in PR #5129: URL: https://github.com/apache/hive/pull/5129#discussion_r1561984544 ## standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java: ## @@ -624,6 +624,40 @@ public boolean openTransaction(String isolationLevel) { return result; } + @Override + public int updateParameterWithExpectedValue(Table table, String key, String expectedValue, String newValue) + throws MetaException { +String dml = String.format( Review Comment: Yes, for old versions where Table object doesn't have the ID, I think we need to retrieve it from `TBLS`. I'm planning to do this in PRs for the release branches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]
chenwyi2 commented on code in PR #5129: URL: https://github.com/apache/hive/pull/5129#discussion_r1561975019 ## standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java: ## @@ -624,6 +624,40 @@ public boolean openTransaction(String isolationLevel) { return result; } + @Override + public int updateParameterWithExpectedValue(Table table, String key, String expectedValue, String newValue) + throws MetaException { +String dml = String.format( Review Comment: is that ok, if i change code as below? ` String queryText = "UPDATE \"TABLE_PARAMS\" SET \"PARAM_VALUE\" = ? " + "WHERE \"TBL_ID\" in (select TBL_ID from TBLS join DBS ON DBS.DB_ID=TBLS.DB_ID WHERE NAME=? and TBL_NAME=? ) AND \"PARAM_KEY\" = ? AND \"PARAM_VALUE\" = ?"; try (QueryWrapper query = new QueryWrapper(pm.newQuery("javax.jdo.query.SQL", queryText))) { Object[] params = new Object[4]; params[0] = ...; long res = query.executeWithArray(params); ... }` cause i don't cherry pick [HIVE-22234](https://issues.apache.org/jira/browse/HIVE-22234), can't get table id -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
sonarcloud[bot] commented on PR #5123: URL: https://github.com/apache/hive/pull/5123#issuecomment-2050724931 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5123) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [19 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5123=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5123=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5123=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5123) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
sonarcloud[bot] commented on PR #5192: URL: https://github.com/apache/hive/pull/5192#issuecomment-2050701347 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5192) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [78 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5192=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5192=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5192=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5192) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] [WIP] Test deserializer for calcite plans [hive]
sonarcloud[bot] commented on PR #5131: URL: https://github.com/apache/hive/pull/5131#issuecomment-2050661598 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5131) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [61 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5131=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5131=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5131=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5131) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] WIP: Dont review [hive]
sonarcloud[bot] commented on PR #5012: URL: https://github.com/apache/hive/pull/5012#issuecomment-2050621836 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5012) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [28 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5012=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5012=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5012=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5012) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-26926: SHOW PARTITIONS for a non partitioned table should just throw execution error instead of full stack trace. [hive]
sonarcloud[bot] commented on PR #5164: URL: https://github.com/apache/hive/pull/5164#issuecomment-2050471702 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5164) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive=5164=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5164=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5164=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5164) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
sonarcloud[bot] commented on PR #5123: URL: https://github.com/apache/hive/pull/5123#issuecomment-2050407048 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5123) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [17 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5123=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5123=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5123=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5123) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] [WIP] Test deserializer for calcite plans [hive]
sonarcloud[bot] commented on PR #5131: URL: https://github.com/apache/hive/pull/5131#issuecomment-2050197847 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5131) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [61 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5131=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5131=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5131=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5131) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
difin commented on code in PR #5123: URL: https://github.com/apache/hive/pull/5123#discussion_r1561329346 ## iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java: ## @@ -21,45 +21,159 @@ import java.io.IOException; import java.util.Map; +import jline.internal.Log; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.conf.HiveConf.ConfVars; +import org.apache.hadoop.hive.metastore.Warehouse; +import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.ql.Context.RewritePolicy; import org.apache.hadoop.hive.ql.DriverUtils; import org.apache.hadoop.hive.ql.metadata.HiveException; +import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec; import org.apache.hadoop.hive.ql.session.SessionState; import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext; import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor; import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName; +import org.apache.iceberg.DataFiles; +import org.apache.iceberg.PartitionData; +import org.apache.iceberg.Table; +import org.apache.iceberg.Transaction; +import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter; +import org.apache.iceberg.mr.hive.IcebergBranchExec; +import org.apache.iceberg.mr.hive.IcebergTableUtil; +import org.apache.iceberg.types.Type; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class IcebergMajorQueryCompactor extends QueryCompactor { private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName()); + private static final String branchName = "compaction"; + private String compactTableName; @Override public boolean run(CompactorContext context) throws IOException, HiveException, InterruptedException { -String compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName()); +compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName()); Map tblProperties = context.getTable().getParameters(); LOG.debug("Initiating compaction for the {} table", compactTableName); -String compactionQuery = String.format("insert overwrite table %s select * from %
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
difin commented on code in PR #5123: URL: https://github.com/apache/hive/pull/5123#discussion_r1561329055 ## iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java: ## @@ -21,45 +21,159 @@ import java.io.IOException; import java.util.Map; +import jline.internal.Log; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.conf.HiveConf.ConfVars; +import org.apache.hadoop.hive.metastore.Warehouse; +import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.ql.Context.RewritePolicy; import org.apache.hadoop.hive.ql.DriverUtils; import org.apache.hadoop.hive.ql.metadata.HiveException; +import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec; import org.apache.hadoop.hive.ql.session.SessionState; import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext; import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor; import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName; +import org.apache.iceberg.DataFiles; +import org.apache.iceberg.PartitionData; +import org.apache.iceberg.Table; +import org.apache.iceberg.Transaction; +import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter; +import org.apache.iceberg.mr.hive.IcebergBranchExec; +import org.apache.iceberg.mr.hive.IcebergTableUtil; +import org.apache.iceberg.types.Type; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class IcebergMajorQueryCompactor extends QueryCompactor { private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName()); + private static final String branchName = "compaction"; Review Comment: done ## iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java: ## @@ -21,45 +21,159 @@ import java.io.IOException; import java.util.Map; +import jline.internal.Log; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.conf.HiveConf.ConfVars; +import org.apache.hadoop.hive.metastore.Warehouse; +import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.ql.Context.RewritePolicy; import org.apache.hadoop.hive.ql.DriverUtils; import org.apache.hadoop.hive.ql.metadata.HiveException; +import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec; import org.apache.hadoop.hive.ql.session.SessionState; import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext; import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor; import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName; +import org.apache.iceberg.DataFiles; +import org.apache.iceberg.PartitionData; +import org.apache.iceberg.Table; +import org.apache.iceberg.Transaction; +import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter; +import org.apache.iceberg.mr.hive.IcebergBranchExec; +import org.apache.iceberg.mr.hive.IcebergTableUtil; +import org.apache.iceberg.types.Type; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class IcebergMajorQueryCompactor extends QueryCompactor { private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName()); + private static final String branchName = "compaction"; + private String compactTableName; Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
difin commented on code in PR #5123: URL: https://github.com/apache/hive/pull/5123#discussion_r1561284855 ## iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java: ## @@ -485,7 +485,7 @@ private void commitTable(FileIO io, ExecutorService executor, OutputTable output } } - private Long getSnapshotId(Table table, String branchName) { + public static Long getSnapshotId(Table table, String branchName) { Review Comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123: URL: https://github.com/apache/hive/pull/5123#discussion_r1561232301 ## iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java: ## @@ -21,45 +21,159 @@ import java.io.IOException; import java.util.Map; +import jline.internal.Log; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.conf.HiveConf.ConfVars; +import org.apache.hadoop.hive.metastore.Warehouse; +import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.ql.Context.RewritePolicy; import org.apache.hadoop.hive.ql.DriverUtils; import org.apache.hadoop.hive.ql.metadata.HiveException; +import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec; import org.apache.hadoop.hive.ql.session.SessionState; import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext; import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor; import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName; +import org.apache.iceberg.DataFiles; +import org.apache.iceberg.PartitionData; +import org.apache.iceberg.Table; +import org.apache.iceberg.Transaction; +import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter; +import org.apache.iceberg.mr.hive.IcebergBranchExec; +import org.apache.iceberg.mr.hive.IcebergTableUtil; +import org.apache.iceberg.types.Type; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class IcebergMajorQueryCompactor extends QueryCompactor { private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName()); + private static final String branchName = "compaction"; + private String compactTableName; @Override public boolean run(CompactorContext context) throws IOException, HiveException, InterruptedException { -String compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName()); +compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName()); Map tblProperties = context.getTable().getParameters(); LOG.debug("Initiating compaction for the {} table", compactTableName); -String compactionQuery = String.format("insert overwrite table %s select * from %
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123: URL: https://github.com/apache/hive/pull/5123#discussion_r1561222370 ## iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java: ## @@ -21,45 +21,159 @@ import java.io.IOException; import java.util.Map; +import jline.internal.Log; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.conf.HiveConf.ConfVars; +import org.apache.hadoop.hive.metastore.Warehouse; +import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.ql.Context.RewritePolicy; import org.apache.hadoop.hive.ql.DriverUtils; import org.apache.hadoop.hive.ql.metadata.HiveException; +import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec; import org.apache.hadoop.hive.ql.session.SessionState; import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext; import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor; import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName; +import org.apache.iceberg.DataFiles; +import org.apache.iceberg.PartitionData; +import org.apache.iceberg.Table; +import org.apache.iceberg.Transaction; +import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter; +import org.apache.iceberg.mr.hive.IcebergBranchExec; +import org.apache.iceberg.mr.hive.IcebergTableUtil; +import org.apache.iceberg.types.Type; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class IcebergMajorQueryCompactor extends QueryCompactor { private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName()); + private static final String branchName = "compaction"; + private String compactTableName; Review Comment: it shouldn't have a state -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123: URL: https://github.com/apache/hive/pull/5123#discussion_r1561226378 ## iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java: ## @@ -21,45 +21,159 @@ import java.io.IOException; import java.util.Map; +import jline.internal.Log; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.conf.HiveConf.ConfVars; +import org.apache.hadoop.hive.metastore.Warehouse; +import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.ql.Context.RewritePolicy; import org.apache.hadoop.hive.ql.DriverUtils; import org.apache.hadoop.hive.ql.metadata.HiveException; +import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec; import org.apache.hadoop.hive.ql.session.SessionState; import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext; import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor; import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName; +import org.apache.iceberg.DataFiles; +import org.apache.iceberg.PartitionData; +import org.apache.iceberg.Table; +import org.apache.iceberg.Transaction; +import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter; +import org.apache.iceberg.mr.hive.IcebergBranchExec; +import org.apache.iceberg.mr.hive.IcebergTableUtil; +import org.apache.iceberg.types.Type; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class IcebergMajorQueryCompactor extends QueryCompactor { private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName()); + private static final String branchName = "compaction"; + private String compactTableName; @Override public boolean run(CompactorContext context) throws IOException, HiveException, InterruptedException { -String compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName()); +compactTableName = TableName.getDbTable(context.getTable().getDbName(), context.getTable().getTableName()); Map tblProperties = context.getTable().getParameters(); LOG.debug("Initiating compaction for the {} table", compactTableName); -String compactionQuery = String.format("insert overwrite table %s select * from %
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123: URL: https://github.com/apache/hive/pull/5123#discussion_r1561221183 ## iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java: ## @@ -21,45 +21,159 @@ import java.io.IOException; import java.util.Map; +import jline.internal.Log; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.conf.HiveConf.ConfVars; +import org.apache.hadoop.hive.metastore.Warehouse; +import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.ql.Context.RewritePolicy; import org.apache.hadoop.hive.ql.DriverUtils; import org.apache.hadoop.hive.ql.metadata.HiveException; +import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec; import org.apache.hadoop.hive.ql.session.SessionState; import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext; import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor; import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName; +import org.apache.iceberg.DataFiles; +import org.apache.iceberg.PartitionData; +import org.apache.iceberg.Table; +import org.apache.iceberg.Transaction; +import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter; +import org.apache.iceberg.mr.hive.IcebergBranchExec; +import org.apache.iceberg.mr.hive.IcebergTableUtil; +import org.apache.iceberg.types.Type; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class IcebergMajorQueryCompactor extends QueryCompactor { private static final Logger LOG = LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName()); + private static final String branchName = "compaction"; Review Comment: finals should be in uppercase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123: URL: https://github.com/apache/hive/pull/5123#discussion_r1561219672 ## iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java: ## @@ -485,7 +485,7 @@ private void commitTable(FileIO io, ExecutorService executor, OutputTable output } } - private Long getSnapshotId(Table table, String branchName) { + public static Long getSnapshotId(Table table, String branchName) { Review Comment: maybe move it to `IcebergTableUtils` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28154: Throw friendly exception if the table does not support partition transform [hive]
deniskuzZ merged PR #5166: URL: https://github.com/apache/hive/pull/5166 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]
lirui-apache commented on PR #5129: URL: https://github.com/apache/hive/pull/5129#issuecomment-2049676147 @deniskuzZ OK, then I'll create PRs for these branches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]
deniskuzZ commented on PR #5129: URL: https://github.com/apache/hive/pull/5129#issuecomment-2049642922 [HIVE-26882](https://issues.apache.org/jira/browse/HIVE-26882) was merged to 3.x and 3.1, so whoever maintains these branches has to cherry-pick -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
kuczoram commented on code in PR #4952: URL: https://github.com/apache/hive/pull/4952#discussion_r1560957112 ## ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java: ## @@ -0,0 +1,313 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hive.ql.txn.compactor; + +import org.apache.commons.lang3.StringUtils; +import org.apache.hadoop.hive.common.StatsSetupConst; +import org.apache.hadoop.hive.metastore.ColumnType; +import org.apache.hadoop.hive.metastore.api.CompactionType; +import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.hadoop.hive.metastore.api.Order; +import org.apache.hadoop.hive.metastore.api.SerDeInfo; +import org.apache.hadoop.hive.metastore.api.SkewedInfo; +import org.apache.hadoop.hive.metastore.api.StorageDescriptor; +import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants; +import org.apache.hadoop.hive.ql.exec.DDLPlanUtils; +import org.apache.hadoop.hive.ql.util.DirectionUtils; +import org.apache.hive.common.util.HiveStringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.lang.reflect.Field; +import java.lang.reflect.Modifier; +import java.util.Arrays; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * Builds query strings that help with query-based compaction of insert-only tables. + */ +class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder { + + private static final Logger LOG = LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName()); + + private StorageDescriptor storageDescriptor; // for Create in insert-only + + /** + * Construct a CompactionQueryBuilderForInsertOnly with required params. + * + * @param compactionType major or minor or rebalance, e.g. CompactionType.MAJOR. + * Cannot be null. + * @param operation query's Operation e.g. Operation.CREATE. + * @throws IllegalArgumentException if compactionType is null or the compaction type is REBALANCE + */ + CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation operation, String resultTableName) { +super(compactionType, operation, true, resultTableName); +if (CompactionType.REBALANCE.equals(compactionType)) { + throw new IllegalArgumentException("Rebalance compaction is supported only on full ACID tables!"); +} + } + + /** + * Set the StorageDescriptor of the table or partition to compact. + * Required for Create operations in insert-only compaction. + * + * @param storageDescriptor StorageDescriptor of the table or partition to compact, not null + */ + CompactionQueryBuilder setStorageDescriptor(StorageDescriptor storageDescriptor) { +this.storageDescriptor = storageDescriptor; +return this; + } + + protected void buildSelectClauseForInsert(StringBuilder query) { Review Comment: Hi @ayushtkn , I am so sorry, I completely missed to response here. Huge thanks for the review! I went through all of your comments and fixed the issues. I just somehow forgot to answer here. As this PR became very old, I opened a new one: https://github.com/apache/hive/pull/5192. That one contains the original changes and the adjustments based on the review comments and also a unit test for the query generation. If you have some time, I would be very happy if you could review it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
[PR] HIVE-27956: Query based compactor implementation separation [hive]
kuczoram opened a new pull request, #5192: URL: https://github.com/apache/hive/pull/5192 Refactored the compaction query generation logic. The main idea was to separate the code parts responsibe for different compaction types. So far everything was handled by one class, therefore it was hard to read and changing the logic for one compaction type might caused side effects on other types. There was an older PR for this change (https://github.com/apache/hive/pull/4952), but it got far away from the master branch, and it was easier to create a new PR. In this PR, I addressed all the comments from the previous one. Also added unit test for the query generation. I ran these tests before the refactoring as well, to be sure that no query output was changed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
kuczoram closed pull request #4952: HIVE-27956: Query based compactor implementation separation URL: https://github.com/apache/hive/pull/4952 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]
kuczoram commented on PR #4952: URL: https://github.com/apache/hive/pull/4952#issuecomment-2049562100 Since this PR is very far from the master now, it is easier to open a new PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28133: Log the original exception in HiveIOExceptionHandlerUtil#handleRecordReaderException [hive]
sonarcloud[bot] commented on PR #5139: URL: https://github.com/apache/hive/pull/5139#issuecomment-2049467602 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5139) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [8 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5139=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5139=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5139=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5139) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]
lirui-apache commented on PR #5129: URL: https://github.com/apache/hive/pull/5129#issuecomment-2049431306 > @lirui-apache: Thanks for the report and the fix @lirui-apache! > > @deniskuzZ: Which branches are likely to get new releases? > > If we want to fix all of the places where the previous PR was released, we might want to add this to: > > * branch-4.0 > * branch-3 > * branch-3.1 > * branch-2 > * branch-2.3 The fix versions of [HIVE-26882](https://issues.apache.org/jira/browse/HIVE-26882) are 2.3.10 and 4.0.0-beta-1. Does that mean we don't need this in 3.x? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]
deniskuzZ commented on PR #5129: URL: https://github.com/apache/hive/pull/5129#issuecomment-2049380680 hey @pvary fix is in, looks good git log --grep "HIVE-28121" we are planning to release 4.0.1 in a month, not sure about other branches -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28154: Throw friendly exception if the table does not support partition transform [hive]
sonarcloud[bot] commented on PR #5166: URL: https://github.com/apache/hive/pull/5166#issuecomment-2049378702 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5166) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [16 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5166=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5166=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5166=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5166) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27165: PART_COL_STATS metastore query not hitting the index [hive]
deniskuzZ commented on code in PR #4141: URL: https://github.com/apache/hive/pull/4141#discussion_r1559718734 ## standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql: ## @@ -101,7 +101,7 @@ CREATE TABLE PART_COL_STATS ALTER TABLE PART_COL_STATS ADD CONSTRAINT PART_COL_STATS_PK PRIMARY KEY (CS_ID); -CREATE INDEX PCS_STATS_IDX ON PART_COL_STATS (CAT_NAME, DB_NAME,TABLE_NAME,COLUMN_NAME,PARTITION_NAME); +CREATE INDEX PCS_STATS_IDX ON PART_COL_STATS (DB_NAME,TABLE_NAME,COLUMN_NAME,PARTITION_NAME, CAT_NAME); Review Comment: order doesn't matter if you use '=' @DanielZhu58 , have you done any benchmarking? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]
pvary commented on PR #5129: URL: https://github.com/apache/hive/pull/5129#issuecomment-2049331261 @lirui-apache: Thanks for the report and the fix @lirui-apache! @deniskuzZ: Which branches are likely to get new releases? If we want to fix all of the places where the previous PR was released, we might want to add this to: - branch-4.0 - branch-3 - branch-3.1 - branch-2 - branch-2.3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28190: Fix heartbeatLockMaterializationRebuild [hive]
deniskuzZ commented on PR #5186: URL: https://github.com/apache/hive/pull/5186#issuecomment-2049271261 please don't merge without the tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28190: Fix heartbeatLockMaterializationRebuild [hive]
kasakrisz commented on PR #5186: URL: https://github.com/apache/hive/pull/5186#issuecomment-2049266842 > @kasakrisz , what do you think, what should be the expected behaviour when the heartbeat fails: > > * should it cancel the query and Hive give an exception, so that the user will know about the issue (I would prefer that way) > * or we can still ignore it. In that case, only a pure luck can help spotting that kind of issues. IMHO It should work like in case of other DML statements (`INSERT SELECT`, `INSERT OVERWRITE SELECT`) and it can be a separate jira/pr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27370: support 4 bytes characters [hive]
sonarcloud[bot] commented on PR #5002: URL: https://github.com/apache/hive/pull/5002#issuecomment-2049262500 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5002) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5002=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5002=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5002=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5002) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
sonarcloud[bot] commented on PR #5123: URL: https://github.com/apache/hive/pull/5123#issuecomment-2049201402 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5123) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [24 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5123=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5123=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5123=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5123) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]
lirui-apache commented on PR #5129: URL: https://github.com/apache/hive/pull/5129#issuecomment-2049197890 Thanks @pvary for reviewing and merging the PR. I'm closing it manually. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]
lirui-apache closed pull request #5129: HIVE-28121: Use direct SQL for transactional altering table parameter URL: https://github.com/apache/hive/pull/5129 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
sonarcloud[bot] commented on PR #5191: URL: https://github.com/apache/hive/pull/5191#issuecomment-2049058390 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5191) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [13 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5191=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5191=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5191=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5191) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] [WIP] Test deserializer for calcite plans [hive]
sonarcloud[bot] commented on PR #5131: URL: https://github.com/apache/hive/pull/5131#issuecomment-2048979041 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5131) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [68 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5131=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5131=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5131=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5131) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org