Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
sonarcloud[bot] commented on PR #4998: URL: https://github.com/apache/hive/pull/4998#issuecomment-2058183246 ## [](https://sonarcloud.io/dashboard?id=apache_hive=4998) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=4998=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=4998=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=4998=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=4998) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
ryukobayashi commented on PR #4998: URL: https://github.com/apache/hive/pull/4998#issuecomment-2058149316 @deniskuzZ I have fixed it as per your suggestion. But, we did not integrate it with UDFLike because it would require major fixes to the logic. Please let me know if you think it should be fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28131: Iceberg: Add support for Replace Branch. [hive]
sonarcloud[bot] commented on PR #5190: URL: https://github.com/apache/hive/pull/5190#issuecomment-2057916449 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5190) **Quality Gate passed** Issues  [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5190=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5190=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5190=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5190) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28131: Iceberg: Add support for Replace Branch. [hive]
ayushtkn commented on code in PR #5190:
URL: https://github.com/apache/hive/pull/5190#discussion_r1566465250
##
parser/src/java/org/apache/hadoop/hive/ql/parse/AlterClauseParser.g:
##
@@ -513,6 +514,13 @@ alterStatementSuffixRenameBranch
-> ^(TOK_ALTERTABLE_RENAME_BRANCH $sourceBranch $targetBranch)
;
+alterStatementSuffixReplaceBranch
+@init { gParent.pushMsg("alter table replace branch", state); }
+@after { gParent.popMsg(state); }
+: KW_REPLACE KW_BRANCH sourceBranch=Identifier KW_WITH KW_SNAPSHOT?
(targetBranch=(Identifier | Number))
Review Comment:
I have copied those from the create branch & pushed a new change, give a
check once if the approach looks fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28131: Iceberg: Add support for Replace Branch. [hive]
ayushtkn commented on code in PR #5190:
URL: https://github.com/apache/hive/pull/5190#discussion_r1566464456
##
iceberg/iceberg-handler/src/test/queries/positive/replace_iceberg_branch.q:
##
@@ -0,0 +1,28 @@
+-- SORT_QUERY_RESULTS
+set hive.explain.user=false;
+set hive.fetch.task.conversion=more;
+
+create external table ice01(id int) stored by iceberg stored as orc
tblproperties ('format-version'='2');
+
+insert into ice01 values (1), (2), (3), (4);
+
+select * from ice01;
+
+-- create one branches
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
[PR] [WIP] Calcite upgrade 1.33 [hive]
soumyakanti3578 opened a new pull request, #5196: URL: https://github.com/apache/hive/pull/5196 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### Is the change a dependency upgrade? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28199: Fix docker issues with Hive 3.1.3 on Mac. [hive]
ryandgoldenberg commented on PR #5178: URL: https://github.com/apache/hive/pull/5178#issuecomment-2057814677 > @ryandgoldenberg Thank you for the contribution, we use jira for issues/case management: https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-JIRA, could you please create one and attach this PR? Thanks in advance! You got it, added [HIVE-28199](https://issues.apache.org/jira/browse/HIVE-28199) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28190: Fix heartbeatLockMaterializationRebuild [hive]
sonarcloud[bot] commented on PR #5186: URL: https://github.com/apache/hive/pull/5186#issuecomment-2057781933 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5186) **Quality Gate passed** Issues  [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive=5186=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5186=new_accepted_issues=list) Measures  [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5186=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5186) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] Draft: ORC projected size fix for ACID [hive]
sonarcloud[bot] commented on PR #5195: URL: https://github.com/apache/hive/pull/5195#issuecomment-2057694125 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5195) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5195=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5195=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5195=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5195) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
[PR] Draft: ORC projected size fix for ACID [hive]
deniskuzZ opened a new pull request, #5195: URL: https://github.com/apache/hive/pull/5195 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### Is the change a dependency upgrade? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565823941
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName,
Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite
commit
*/
private void commitOverwrite(Table table, String branchName, long startTime,
FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not
handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+LOG.info("Overwrite commit: truncated all partitions for table: {}
with {} file(s)",
+table, results.dataFiles().size());
+ } else if (!partitionSpec.isEmpty()) {
Review Comment:
This is not for partition evolution case. This is for compaction on
partition level without evolution. The partition needs to be truncated before
adding compacted files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28196: Preserve column stats when applying UDF upper/lower [hive]
okumin commented on code in PR #5191:
URL: https://github.com/apache/hive/pull/5191#discussion_r1565852758
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -1736,6 +1749,13 @@ private static boolean
isWideningCast(ExprNodeGenericFuncDesc engfd) {
engfd.getTypeInfo());
}
+ private static boolean isColStatsReusableFunc(ExprNodeGenericFuncDesc engfd)
{
+GenericUDF udf = engfd.getGenericUDF();
+Class udfClass =
+(udf instanceof GenericUDFBridge) ?
((GenericUDFBridge)udf).getUdfClass() : udf.getClass();
+return udfClass == GenericUDFLower.class || udfClass ==
GenericUDFUpper.class;
Review Comment:
I like this PR. I have also observed that an estimation was wrong, and Map
Join was disabled.
We implement some estimation APIs on the UDF side. I'll share examples with
you. This is not a strong suggestion, and the current implementation seems to
be a good starting point. We may finally need a pluggable I/F to empower custom
UDFs more.
-
https://github.com/apache/hive/blob/rel/release-4.0.0/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NDV.java
-
https://github.com/apache/hive/blob/rel/release-4.0.0/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java#L63-L81
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] Fix docker issues with Hive 3.1.3 on Mac. [hive]
dengzhhu653 commented on PR #5178: URL: https://github.com/apache/hive/pull/5178#issuecomment-2056917417 @ryandgoldenberg Thank you for the contribution, we use jira for issues/case management: https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-JIRA, could you please create one and attach this PR? Thanks in advance! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565823941
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName,
Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite
commit
*/
private void commitOverwrite(Table table, String branchName, long startTime,
FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not
handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+LOG.info("Overwrite commit: truncated all partitions for table: {}
with {} file(s)",
+table, results.dataFiles().size());
+ } else if (!partitionSpec.isEmpty()) {
Review Comment:
This is not for partition evolution case. This is for compaction on
partition level without evolution. The partition needs to be truncated before
adding compacted files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] Update Hive version in Docker README [hive]
sonarcloud[bot] commented on PR #5105: URL: https://github.com/apache/hive/pull/5105#issuecomment-2056888797 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5105) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5105=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5105=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5105=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5105) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] Update Hive version in Docker README [hive]
dengzhhu653 commented on code in PR #5105: URL: https://github.com/apache/hive/pull/5105#discussion_r1565807011 ## packaging/src/docker/README.md: ## @@ -59,7 +60,7 @@ If the version is not provided, it will read the version from current `pom.xml`: For example, the following command uses Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` to build the image, Review Comment: Change `uses Hive 3.1.3` to `Hive 4.0.0`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] Update Hive version in Docker README [hive]
mertyyanik commented on code in PR #5105: URL: https://github.com/apache/hive/pull/5105#discussion_r1565745559 ## packaging/src/docker/README.md: ## @@ -5,8 +5,39 @@ Run Apache Hive inside docker container in pseudo-distributed mode, provide the - Quick-start/Debugging/Prepare a test env for Hive -### Quickstart +## Quickstart +### STEP 1: Pull the image +- Pull the image from DockerHub: https://hub.docker.com/r/apache/hive/tags. +Here are the latest images: +- 4.0.0-beta-1 Review Comment: I updated the hive version, but where exactly should I update the hadoop and tez version? @deniskuzZ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] Update Hive version in Docker README [hive]
mertyyanik commented on code in PR #5105: URL: https://github.com/apache/hive/pull/5105#discussion_r1565745559 ## packaging/src/docker/README.md: ## @@ -5,8 +5,39 @@ Run Apache Hive inside docker container in pseudo-distributed mode, provide the - Quick-start/Debugging/Prepare a test env for Hive -### Quickstart +## Quickstart +### STEP 1: Pull the image +- Pull the image from DockerHub: https://hub.docker.com/r/apache/hive/tags. +Here are the latest images: +- 4.0.0-beta-1 Review Comment: I updated the hive version, but where exactly should I update the hadoop and tez version? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] Update Hive version in Docker README [hive]
mertyyanik commented on code in PR #5105: URL: https://github.com/apache/hive/pull/5105#discussion_r1565745559 ## packaging/src/docker/README.md: ## @@ -5,8 +5,39 @@ Run Apache Hive inside docker container in pseudo-distributed mode, provide the - Quick-start/Debugging/Prepare a test env for Hive -### Quickstart +## Quickstart +### STEP 1: Pull the image +- Pull the image from DockerHub: https://hub.docker.com/r/apache/hive/tags. +Here are the latest images: +- 4.0.0-beta-1 Review Comment: I updated the hive version, but where exactly should I update the hadoop and thesis version? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565744058
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##
@@ -44,22 +52,68 @@ public boolean run(CompactorContext context) throws
IOException, HiveException,
Map tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
-String compactionQuery = String.format("insert overwrite table %s select *
from % partSpecMap = Warehouse.makeSpecFromName(partSpec);
+ StringBuilder sb = new StringBuilder();
+
+ for (int i = 0; i < partitionData.size(); ++i) {
+if (i > 0) {
+ sb.append(delimiter);
+}
+
+String quoteOpt = "";
+if (partitionData.getType(i).typeId() == Type.TypeID.STRING ||
+partitionData.getType(i).typeId() == Type.TypeID.DATE ||
+partitionData.getType(i).typeId() == Type.TypeID.TIME ||
+partitionData.getType(i).typeId() == Type.TypeID.TIMESTAMP ||
+partitionData.getType(i).typeId() == Type.TypeID.BINARY) {
+ quoteOpt = "'";
+}
+
+sb.append(partitionData.getSchema().getFields().get(i).name())
+.append("=")
+.append(quoteOpt)
+
.append(partSpecMap.get(partitionData.getPartitionType().fields().get(i).name()))
Review Comment:
should we use `partitionData.getSchema().getFields()` here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565481530
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName,
Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite
commit
*/
private void commitOverwrite(Table table, String branchName, long startTime,
FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not
handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+LOG.info("Overwrite commit: truncated all partitions for table: {}
with {} file(s)",
+table, results.dataFiles().size());
+ } else if (!partitionSpec.isEmpty()) {
Review Comment:
why do we need this? that won't help with partition evolution in any case
ATM (no evolution support) partitionSpec is not required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565694331
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##
@@ -44,22 +52,68 @@ public boolean run(CompactorContext context) throws
IOException, HiveException,
Map tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
-String compactionQuery = String.format("insert overwrite table %s select *
from %
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565694331
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##
@@ -44,22 +52,68 @@ public boolean run(CompactorContext context) throws
IOException, HiveException,
Map tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
-String compactionQuery = String.format("insert overwrite table %s select *
from %
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565694331
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##
@@ -44,22 +52,68 @@ public boolean run(CompactorContext context) throws
IOException, HiveException,
Map tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
-String compactionQuery = String.format("insert overwrite table %s select *
from %
Re: [PR] HIVE-28197: Add deserializer to convert JSON plans to RelNodes [hive]
sonarcloud[bot] commented on PR #5131: URL: https://github.com/apache/hive/pull/5131#issuecomment-2056526306 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5131) **Quality Gate passed** Issues  [59 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5131=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5131=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5131=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5131) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28137: Append Multi-Level Partition Filtering in Direct SQL Implementation of getPartitionsByFilter [hive]
cxzl25 commented on PR #5141: URL: https://github.com/apache/hive/pull/5141#issuecomment-2056452806 @saihemanth-cloudera @zhangbutao Please help review again when you have time. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28153: Flaky test TestConflictingDataFiles.testMultiFiltersUpdate [hive]
simhadri-g commented on PR #5193: URL: https://github.com/apache/hive/pull/5193#issuecomment-2056452513 Thanks @deniskuzZ and @zhangbutao for the review :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565481530
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName,
Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite
commit
*/
private void commitOverwrite(Table table, String branchName, long startTime,
FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not
handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
DeleteFiles delete = transaction.newDelete();
delete.deleteFromRowFilter(Expressions.alwaysTrue());
delete.commit();
+LOG.info("Overwrite commit: truncated all partitions for table: {}
with {} file(s)",
+table, results.dataFiles().size());
+ } else if (!partitionSpec.isEmpty()) {
Review Comment:
why do we need this? that won't help with partition evolution in any case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565474973
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName,
Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite
commit
*/
private void commitOverwrite(Table table, String branchName, long startTime,
FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not
handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
Review Comment:
do we need a rewrite policy, since we can check for `partitionSpec.isEmpty()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565474973
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -569,14 +575,25 @@ private void commitWrite(Table table, String branchName,
Long snapshotId, long s
* @param rewritePolicy The rewrite policy to use for the insert overwrite
commit
*/
private void commitOverwrite(Table table, String branchName, long startTime,
FilesForCommit results,
- RewritePolicy rewritePolicy) {
+ RewritePolicy rewritePolicy, String partitionSpec) {
Preconditions.checkArgument(results.deleteFiles().isEmpty(), "Can not
handle deletes with overwrite");
if (!results.dataFiles().isEmpty()) {
Transaction transaction = table.newTransaction();
if (rewritePolicy == RewritePolicy.ALL_PARTITIONS) {
Review Comment:
do we need a rewrite policy, since we can check for `partitionSpec.isEmpty()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28077: Iceberg: Major QB Compaction on partition level [hive]
deniskuzZ commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1565469401
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##
@@ -640,7 +640,13 @@ public void
preTruncateTable(org.apache.hadoop.hive.metastore.api.Table table, E
throws MetaException {
this.catalogProperties = getCatalogProperties(table);
this.icebergTable = Catalogs.loadTable(conf, catalogProperties);
-Map partitionFieldMap =
icebergTable.spec().fields().stream()
+truncatePartitionBySpec(icebergTable, partNames, icebergTable.newDelete());
+context.putToProperties("truncateSkipDataDeletion", "true");
Review Comment:
i don't see constant used here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28190: Fix heartbeatLockMaterializationRebuild [hive]
zratkai commented on code in PR #5186:
URL: https://github.com/apache/hive/pull/5186#discussion_r1565469449
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -705,12 +706,11 @@ public LockResponse lockMaterializationRebuild(String
dbName, String tableName,
public boolean heartbeatLockMaterializationRebuild(String dbName, String
tableName, long txnId) throws MetaException {
int result = jdbcResource.execute(
"UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-" SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-" WHERE \"MRL_TXN_ID\" = " + txnId +
-" AND \"MRL_DB_NAME\" = ?" +
-" AND \"MRL_TBL_NAME\" = ?",
+" SET \"MRL_LAST_HEARTBEAT\" = " +
getEpochFn(jdbcResource.getDatabaseProduct()) +
Review Comment:
Instead of concatenating the timestamp, I would prefer to use it as
parametrised value like txnId, dbName.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28133: Log the original exception in HiveIOExceptionHandlerUtil#handleRecordReaderException [hive]
deniskuzZ merged PR #5139: URL: https://github.com/apache/hive/pull/5139 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28197: Add deserializer to convert JSON plans to RelNodes [hive]
sonarcloud[bot] commented on PR #5131: URL: https://github.com/apache/hive/pull/5131#issuecomment-2055794675 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5131) **Quality Gate passed** Issues  [59 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5131=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5131=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5131=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5131) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] Testing [hive]
sonarcloud[bot] commented on PR #5194: URL: https://github.com/apache/hive/pull/5194#issuecomment-205349 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5194) **Quality Gate passed** Issues  [20 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5194=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive=5194=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5194=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5194) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
