Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]

2024-04-11 Thread via GitHub


lirui-apache commented on code in PR #5129:
URL: https://github.com/apache/hive/pull/5129#discussion_r1561984544


##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##
@@ -624,6 +624,40 @@ public boolean openTransaction(String isolationLevel) {
 return result;
   }
 
+  @Override
+  public int updateParameterWithExpectedValue(Table table, String key, String 
expectedValue, String newValue)
+  throws MetaException {
+String dml = String.format(

Review Comment:
   Yes, for old versions where Table object doesn't have the ID, I think we 
need to retrieve it from `TBLS`. I'm planning to do this in PRs for the release 
branches.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]

2024-04-11 Thread via GitHub


chenwyi2 commented on code in PR #5129:
URL: https://github.com/apache/hive/pull/5129#discussion_r1561975019


##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##
@@ -624,6 +624,40 @@ public boolean openTransaction(String isolationLevel) {
 return result;
   }
 
+  @Override
+  public int updateParameterWithExpectedValue(Table table, String key, String 
expectedValue, String newValue)
+  throws MetaException {
+String dml = String.format(

Review Comment:
   is that ok, if i change code as below?
   `   String queryText =  "UPDATE \"TABLE_PARAMS\" SET \"PARAM_VALUE\" = ? " +
   "WHERE \"TBL_ID\" in (select TBL_ID from TBLS join DBS ON 
DBS.DB_ID=TBLS.DB_ID WHERE NAME=? and TBL_NAME=? ) AND \"PARAM_KEY\" = ? AND 
\"PARAM_VALUE\" = ?";
   try (QueryWrapper query = new 
QueryWrapper(pm.newQuery("javax.jdo.query.SQL", queryText))) {
  Object[] params = new Object[4];
  params[0] = ...;
  long res = query.executeWithArray(params);
   ...
   }`
   cause i don't cherry pick 
[HIVE-22234](https://issues.apache.org/jira/browse/HIVE-22234), can't get table 
id



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2050724931

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5123) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [19 New 
issues](https://sonarcloud.io/project/issues?id=apache_hive=5123=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5123=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5123=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5123)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5192:
URL: https://github.com/apache/hive/pull/5192#issuecomment-2050701347

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5192) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [78 New 
issues](https://sonarcloud.io/project/issues?id=apache_hive=5192=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5192=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5192=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5192)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] [WIP] Test deserializer for calcite plans [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5131:
URL: https://github.com/apache/hive/pull/5131#issuecomment-2050661598

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5131) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [61 New 
issues](https://sonarcloud.io/project/issues?id=apache_hive=5131=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5131=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5131=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5131)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] WIP: Dont review [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5012:
URL: https://github.com/apache/hive/pull/5012#issuecomment-2050621836

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5012) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [28 New 
issues](https://sonarcloud.io/project/issues?id=apache_hive=5012=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5012=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5012=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5012)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-26926: SHOW PARTITIONS for a non partitioned table should just throw execution error instead of full stack trace. [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5164:
URL: https://github.com/apache/hive/pull/5164#issuecomment-2050471702

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5164) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [1 New 
issue](https://sonarcloud.io/project/issues?id=apache_hive=5164=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5164=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5164=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5164)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2050407048

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5123) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [17 New 
issues](https://sonarcloud.io/project/issues?id=apache_hive=5123=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5123=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5123=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5123)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] [WIP] Test deserializer for calcite plans [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5131:
URL: https://github.com/apache/hive/pull/5131#issuecomment-2050197847

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5131) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [61 New 
issues](https://sonarcloud.io/project/issues?id=apache_hive=5131=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5131=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5131=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5131)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]

2024-04-11 Thread via GitHub


difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561329346


##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##
@@ -21,45 +21,159 @@
 
 import java.io.IOException;
 import java.util.Map;
+import jline.internal.Log;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
 import org.apache.hadoop.hive.ql.Context.RewritePolicy;
 import org.apache.hadoop.hive.ql.DriverUtils;
 import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
 import org.apache.hadoop.hive.ql.session.SessionState;
 import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
 import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
 import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 public class IcebergMajorQueryCompactor extends QueryCompactor  {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+  private static final String branchName = "compaction";
+  private String compactTableName;
 
   @Override
   public boolean run(CompactorContext context) throws IOException, 
HiveException, InterruptedException {
 
-String compactTableName = 
TableName.getDbTable(context.getTable().getDbName(), 
context.getTable().getTableName());
+compactTableName = TableName.getDbTable(context.getTable().getDbName(), 
context.getTable().getTableName());
 Map tblProperties = context.getTable().getParameters();
 LOG.debug("Initiating compaction for the {} table", compactTableName);
 
-String compactionQuery = String.format("insert overwrite table %s select * 
from %

Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]

2024-04-11 Thread via GitHub


difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561329055


##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##
@@ -21,45 +21,159 @@
 
 import java.io.IOException;
 import java.util.Map;
+import jline.internal.Log;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
 import org.apache.hadoop.hive.ql.Context.RewritePolicy;
 import org.apache.hadoop.hive.ql.DriverUtils;
 import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
 import org.apache.hadoop.hive.ql.session.SessionState;
 import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
 import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
 import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 public class IcebergMajorQueryCompactor extends QueryCompactor  {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+  private static final String branchName = "compaction";

Review Comment:
   done



##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##
@@ -21,45 +21,159 @@
 
 import java.io.IOException;
 import java.util.Map;
+import jline.internal.Log;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
 import org.apache.hadoop.hive.ql.Context.RewritePolicy;
 import org.apache.hadoop.hive.ql.DriverUtils;
 import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
 import org.apache.hadoop.hive.ql.session.SessionState;
 import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
 import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
 import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 public class IcebergMajorQueryCompactor extends QueryCompactor  {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+  private static final String branchName = "compaction";
+  private String compactTableName;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]

2024-04-11 Thread via GitHub


difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561284855


##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -485,7 +485,7 @@ private void commitTable(FileIO io, ExecutorService 
executor, OutputTable output
 }
   }
 
-  private Long getSnapshotId(Table table, String branchName) {
+  public static Long getSnapshotId(Table table, String branchName) {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]

2024-04-11 Thread via GitHub


deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561232301


##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##
@@ -21,45 +21,159 @@
 
 import java.io.IOException;
 import java.util.Map;
+import jline.internal.Log;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
 import org.apache.hadoop.hive.ql.Context.RewritePolicy;
 import org.apache.hadoop.hive.ql.DriverUtils;
 import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
 import org.apache.hadoop.hive.ql.session.SessionState;
 import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
 import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
 import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 public class IcebergMajorQueryCompactor extends QueryCompactor  {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+  private static final String branchName = "compaction";
+  private String compactTableName;
 
   @Override
   public boolean run(CompactorContext context) throws IOException, 
HiveException, InterruptedException {
 
-String compactTableName = 
TableName.getDbTable(context.getTable().getDbName(), 
context.getTable().getTableName());
+compactTableName = TableName.getDbTable(context.getTable().getDbName(), 
context.getTable().getTableName());
 Map tblProperties = context.getTable().getParameters();
 LOG.debug("Initiating compaction for the {} table", compactTableName);
 
-String compactionQuery = String.format("insert overwrite table %s select * 
from %

Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]

2024-04-11 Thread via GitHub


deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561222370


##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##
@@ -21,45 +21,159 @@
 
 import java.io.IOException;
 import java.util.Map;
+import jline.internal.Log;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
 import org.apache.hadoop.hive.ql.Context.RewritePolicy;
 import org.apache.hadoop.hive.ql.DriverUtils;
 import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
 import org.apache.hadoop.hive.ql.session.SessionState;
 import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
 import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
 import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 public class IcebergMajorQueryCompactor extends QueryCompactor  {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+  private static final String branchName = "compaction";
+  private String compactTableName;

Review Comment:
   it shouldn't have a state



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]

2024-04-11 Thread via GitHub


deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561226378


##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##
@@ -21,45 +21,159 @@
 
 import java.io.IOException;
 import java.util.Map;
+import jline.internal.Log;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
 import org.apache.hadoop.hive.ql.Context.RewritePolicy;
 import org.apache.hadoop.hive.ql.DriverUtils;
 import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
 import org.apache.hadoop.hive.ql.session.SessionState;
 import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
 import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
 import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 public class IcebergMajorQueryCompactor extends QueryCompactor  {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+  private static final String branchName = "compaction";
+  private String compactTableName;
 
   @Override
   public boolean run(CompactorContext context) throws IOException, 
HiveException, InterruptedException {
 
-String compactTableName = 
TableName.getDbTable(context.getTable().getDbName(), 
context.getTable().getTableName());
+compactTableName = TableName.getDbTable(context.getTable().getDbName(), 
context.getTable().getTableName());
 Map tblProperties = context.getTable().getParameters();
 LOG.debug("Initiating compaction for the {} table", compactTableName);
 
-String compactionQuery = String.format("insert overwrite table %s select * 
from %

Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]

2024-04-11 Thread via GitHub


deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561221183


##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##
@@ -21,45 +21,159 @@
 
 import java.io.IOException;
 import java.util.Map;
+import jline.internal.Log;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
 import org.apache.hadoop.hive.ql.Context.RewritePolicy;
 import org.apache.hadoop.hive.ql.DriverUtils;
 import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.AlterTableSnapshotRefSpec;
 import org.apache.hadoop.hive.ql.session.SessionState;
 import org.apache.hadoop.hive.ql.txn.compactor.CompactorContext;
 import org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor;
 import org.apache.hive.iceberg.org.apache.orc.storage.common.TableName;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.PartitionData;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter;
+import org.apache.iceberg.mr.hive.IcebergBranchExec;
+import org.apache.iceberg.mr.hive.IcebergTableUtil;
+import org.apache.iceberg.types.Type;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 public class IcebergMajorQueryCompactor extends QueryCompactor  {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(IcebergMajorQueryCompactor.class.getName());
+  private static final String branchName = "compaction";

Review Comment:
   finals should be in uppercase



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]

2024-04-11 Thread via GitHub


deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1561219672


##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -485,7 +485,7 @@ private void commitTable(FileIO io, ExecutorService 
executor, OutputTable output
 }
   }
 
-  private Long getSnapshotId(Table table, String branchName) {
+  public static Long getSnapshotId(Table table, String branchName) {

Review Comment:
   maybe move it to `IcebergTableUtils`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28154: Throw friendly exception if the table does not support partition transform [hive]

2024-04-11 Thread via GitHub


deniskuzZ merged PR #5166:
URL: https://github.com/apache/hive/pull/5166


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]

2024-04-11 Thread via GitHub


lirui-apache commented on PR #5129:
URL: https://github.com/apache/hive/pull/5129#issuecomment-2049676147

   @deniskuzZ OK, then I'll create PRs for these branches.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]

2024-04-11 Thread via GitHub


deniskuzZ commented on PR #5129:
URL: https://github.com/apache/hive/pull/5129#issuecomment-2049642922

   [HIVE-26882](https://issues.apache.org/jira/browse/HIVE-26882) was merged to 
3.x and 3.1, so whoever maintains these branches has to cherry-pick


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]

2024-04-11 Thread via GitHub


kuczoram commented on code in PR #4952:
URL: https://github.com/apache/hive/pull/4952#discussion_r1560957112


##
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactionQueryBuilderForInsertOnly.java:
##
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.exec.DDLPlanUtils;
+import org.apache.hadoop.hive.ql.util.DirectionUtils;
+import org.apache.hive.common.util.HiveStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Builds query strings that help with query-based compaction of insert-only 
tables.
+ */
+class CompactionQueryBuilderForInsertOnly extends CompactionQueryBuilder {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(CompactionQueryBuilderForInsertOnly.class.getName());
+
+  private StorageDescriptor storageDescriptor; // for Create in insert-only
+
+  /**
+   * Construct a CompactionQueryBuilderForInsertOnly with required params.
+   *
+   * @param compactionType major or minor or rebalance, e.g. 
CompactionType.MAJOR.
+   *   Cannot be null.
+   * @param operation  query's Operation e.g. Operation.CREATE.
+   * @throws IllegalArgumentException if compactionType is null or the 
compaction type is REBALANCE
+   */
+  CompactionQueryBuilderForInsertOnly(CompactionType compactionType, Operation 
operation, String resultTableName) {
+super(compactionType, operation, true, resultTableName);
+if (CompactionType.REBALANCE.equals(compactionType)) {
+  throw new IllegalArgumentException("Rebalance compaction is supported 
only on full ACID tables!");
+}
+  }
+
+  /**
+   * Set the StorageDescriptor of the table or partition to compact.
+   * Required for Create operations in insert-only compaction.
+   *
+   * @param storageDescriptor StorageDescriptor of the table or partition to 
compact, not null
+   */
+  CompactionQueryBuilder setStorageDescriptor(StorageDescriptor 
storageDescriptor) {
+this.storageDescriptor = storageDescriptor;
+return this;
+  }
+
+  protected void buildSelectClauseForInsert(StringBuilder query) {

Review Comment:
   Hi @ayushtkn ,
   I am so sorry, I completely missed to response here. Huge thanks for the 
review! I went through all of your comments and fixed the issues. I just 
somehow forgot to answer here. As this PR became very old, I opened a new one: 
https://github.com/apache/hive/pull/5192. That one contains the original 
changes and the adjustments based on the review comments and also a unit test 
for the query generation. If you have some time, I would be very happy if you 
could review it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



[PR] HIVE-27956: Query based compactor implementation separation [hive]

2024-04-11 Thread via GitHub


kuczoram opened a new pull request, #5192:
URL: https://github.com/apache/hive/pull/5192

   Refactored the compaction query generation logic. The main idea was to 
separate the code parts responsibe for different compaction types. So far 
everything was handled by one class, therefore it was hard to read and changing 
the logic for one compaction type might caused side effects on other types. 
   
   There was an older PR for this change 
(https://github.com/apache/hive/pull/4952), but it got far away from the master 
branch, and it was easier to create a new PR.
   In this PR, I addressed all the comments from the previous one.
   Also added unit test for the query generation. I ran these tests before the 
refactoring as well, to be sure that no query output was changed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]

2024-04-11 Thread via GitHub


kuczoram closed pull request #4952: HIVE-27956: Query based compactor 
implementation separation
URL: https://github.com/apache/hive/pull/4952


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-27956: Query based compactor implementation separation [hive]

2024-04-11 Thread via GitHub


kuczoram commented on PR #4952:
URL: https://github.com/apache/hive/pull/4952#issuecomment-2049562100

   Since this PR is very far from the master now, it is easier to open a new PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28133: Log the original exception in HiveIOExceptionHandlerUtil#handleRecordReaderException [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5139:
URL: https://github.com/apache/hive/pull/5139#issuecomment-2049467602

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5139) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [8 New 
issues](https://sonarcloud.io/project/issues?id=apache_hive=5139=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5139=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5139=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5139)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]

2024-04-11 Thread via GitHub


lirui-apache commented on PR #5129:
URL: https://github.com/apache/hive/pull/5129#issuecomment-2049431306

   > @lirui-apache: Thanks for the report and the fix @lirui-apache!
   > 
   > @deniskuzZ: Which branches are likely to get new releases?
   > 
   > If we want to fix all of the places where the previous PR was released, we 
might want to add this to:
   > 
   > * branch-4.0
   > * branch-3
   > * branch-3.1
   > * branch-2
   > * branch-2.3
   
   The fix versions of 
[HIVE-26882](https://issues.apache.org/jira/browse/HIVE-26882) are 2.3.10 and 
4.0.0-beta-1. Does that mean we don't need this in 3.x?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]

2024-04-11 Thread via GitHub


deniskuzZ commented on PR #5129:
URL: https://github.com/apache/hive/pull/5129#issuecomment-2049380680

   hey @pvary 
   fix is in, looks good
   
   git log --grep "HIVE-28121"
   
   we are planning to release 4.0.1 in a month, not sure about other branches


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28154: Throw friendly exception if the table does not support partition transform [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5166:
URL: https://github.com/apache/hive/pull/5166#issuecomment-2049378702

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5166) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [16 New 
issues](https://sonarcloud.io/project/issues?id=apache_hive=5166=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5166=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5166=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5166)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-27165: PART_COL_STATS metastore query not hitting the index [hive]

2024-04-11 Thread via GitHub


deniskuzZ commented on code in PR #4141:
URL: https://github.com/apache/hive/pull/4141#discussion_r1559718734


##
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql:
##
@@ -101,7 +101,7 @@ CREATE TABLE PART_COL_STATS
 
 ALTER TABLE PART_COL_STATS ADD CONSTRAINT PART_COL_STATS_PK PRIMARY KEY 
(CS_ID);
 
-CREATE INDEX PCS_STATS_IDX ON PART_COL_STATS (CAT_NAME, 
DB_NAME,TABLE_NAME,COLUMN_NAME,PARTITION_NAME);
+CREATE INDEX PCS_STATS_IDX ON PART_COL_STATS 
(DB_NAME,TABLE_NAME,COLUMN_NAME,PARTITION_NAME, CAT_NAME);

Review Comment:
   order doesn't matter if you use '='
   @DanielZhu58 , have you done any benchmarking?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]

2024-04-11 Thread via GitHub


pvary commented on PR #5129:
URL: https://github.com/apache/hive/pull/5129#issuecomment-2049331261

   @lirui-apache: Thanks for the report and the fix @lirui-apache!
   
   @deniskuzZ: Which branches are likely to get new releases?
   
   If we want to fix all of the places where the previous PR was released, we 
might want to add this to:
   - branch-4.0
   - branch-3
   - branch-3.1
   - branch-2
   - branch-2.3
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28190: Fix heartbeatLockMaterializationRebuild [hive]

2024-04-11 Thread via GitHub


deniskuzZ commented on PR #5186:
URL: https://github.com/apache/hive/pull/5186#issuecomment-2049271261

   please don't merge without the tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28190: Fix heartbeatLockMaterializationRebuild [hive]

2024-04-11 Thread via GitHub


kasakrisz commented on PR #5186:
URL: https://github.com/apache/hive/pull/5186#issuecomment-2049266842

   > @kasakrisz , what do you think, what should be the expected behaviour when 
the heartbeat fails:
   > 
   > * should it cancel the query and Hive give an exception, so that the user 
will know about the issue (I would prefer that way)
   > * or we can still ignore it. In that case, only a pure luck can help 
spotting that kind of issues.
   
   
   IMHO It should work like in case of other DML statements (`INSERT SELECT`, 
`INSERT OVERWRITE SELECT`) and it can be a separate jira/pr


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-27370: support 4 bytes characters [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5002:
URL: https://github.com/apache/hive/pull/5002#issuecomment-2049262500

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5002) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [10 New 
issues](https://sonarcloud.io/project/issues?id=apache_hive=5002=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5002=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5002=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5002)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5123:
URL: https://github.com/apache/hive/pull/5123#issuecomment-2049201402

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5123) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [24 New 
issues](https://sonarcloud.io/project/issues?id=apache_hive=5123=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5123=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5123=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5123)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]

2024-04-11 Thread via GitHub


lirui-apache commented on PR #5129:
URL: https://github.com/apache/hive/pull/5129#issuecomment-2049197890

   Thanks @pvary for reviewing and merging the PR. I'm closing it manually.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28121: Use direct SQL for transactional altering table parameter [hive]

2024-04-11 Thread via GitHub


lirui-apache closed pull request #5129: HIVE-28121: Use direct SQL for 
transactional altering table parameter
URL: https://github.com/apache/hive/pull/5129


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5191:
URL: https://github.com/apache/hive/pull/5191#issuecomment-2049058390

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5191) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [13 New 
issues](https://sonarcloud.io/project/issues?id=apache_hive=5191=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5191=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5191=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5191)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org



Re: [PR] [WIP] Test deserializer for calcite plans [hive]

2024-04-11 Thread via GitHub


sonarcloud[bot] commented on PR #5131:
URL: https://github.com/apache/hive/pull/5131#issuecomment-2048979041

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_hive=5131) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [68 New 
issues](https://sonarcloud.io/project/issues?id=apache_hive=5131=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_hive=5131=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5131=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5131)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org