Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
codope merged PR #12280: URL: https://github.com/apache/hudi/pull/12280 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2490411785 ## CI report: * 18af42ffaec1ae1032b71219e89947ae2d0c0e92 UNKNOWN * 9661618b7d8c4d2521b80b37d798773276f2e403 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1800) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2490283483 ## CI report: * 18af42ffaec1ae1032b71219e89947ae2d0c0e92 UNKNOWN * f128cc5b22df3ced6a7cc8b0dee0040f1e35034a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1788) * 9661618b7d8c4d2521b80b37d798773276f2e403 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1800) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
codope commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2490264696 I've fixed the test failures. Should be good to land after they pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2490245561 ## CI report: * 18af42ffaec1ae1032b71219e89947ae2d0c0e92 UNKNOWN * f128cc5b22df3ced6a7cc8b0dee0040f1e35034a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1788) * 9661618b7d8c4d2521b80b37d798773276f2e403 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
nsivabalan commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2489968603 there are some test failures. can you check it out. @lokeshj1703 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2489487559 ## CI report: * 18af42ffaec1ae1032b71219e89947ae2d0c0e92 UNKNOWN * f128cc5b22df3ced6a7cc8b0dee0040f1e35034a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1788) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2489389950 ## CI report: * 7fe917a9e87e95a3422775faf1931ef1b9b188e4 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1785) * 18af42ffaec1ae1032b71219e89947ae2d0c0e92 UNKNOWN * f128cc5b22df3ced6a7cc8b0dee0040f1e35034a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1788) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2489377887 ## CI report: * 7fe917a9e87e95a3422775faf1931ef1b9b188e4 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1785) * 18af42ffaec1ae1032b71219e89947ae2d0c0e92 UNKNOWN * f128cc5b22df3ced6a7cc8b0dee0040f1e35034a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2489345090 ## CI report: * 7fe917a9e87e95a3422775faf1931ef1b9b188e4 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1785) * 18af42ffaec1ae1032b71219e89947ae2d0c0e92 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2489341681 ## CI report: * c3a4cabc61d92019af39d6a23a90c8f8594f46d0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1774) * 7fe917a9e87e95a3422775faf1931ef1b9b188e4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1785) * 18af42ffaec1ae1032b71219e89947ae2d0c0e92 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
codope commented on code in PR #12280: URL: https://github.com/apache/hudi/pull/12280#discussion_r1850839700 ## hudi-client/hudi-java-client/src/test/java/org/apache/hudi/testutils/HoodieJavaClientTestHarness.java: ## @@ -429,22 +428,12 @@ private void runFullValidation(HoodieWriteConfig writeConfig, List metadataTablePartitions = FSUtils.getAllPartitionPaths( engineContext, storage, HoodieTableMetadata.getMetadataTableBasePath(basePath), false); - List enabledPartitionTypes = metadataWriter.getEnabledPartitionTypes(); - - assertEquals(enabledPartitionTypes.size(), metadataTablePartitions.size()); Review Comment: For secondary and functional indexes, enabledPartitionTypes do not necessarily mean available partitions because secondary and functional index partition type can have multiple metadata partitions. This assertion does not add much value, hence removing it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2489283413 ## CI report: * c3a4cabc61d92019af39d6a23a90c8f8594f46d0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1774) * 7fe917a9e87e95a3422775faf1931ef1b9b188e4 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2489286742 ## CI report: * c3a4cabc61d92019af39d6a23a90c8f8594f46d0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1774) * 7fe917a9e87e95a3422775faf1931ef1b9b188e4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1785) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
codope commented on code in PR #12280:
URL: https://github.com/apache/hudi/pull/12280#discussion_r1850797043
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestSecondaryIndex.scala:
##
@@ -284,6 +286,50 @@ class TestSecondaryIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Secondary Index With Overwrite and Delete Partition") {
+if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+Seq(
+ WriteOperationType.INSERT_OVERWRITE.value(),
+ WriteOperationType.INSERT_OVERWRITE_TABLE.value(),
+ WriteOperationType.DELETE_PARTITION.value()
+).foreach { operationType =>
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ // Step 1: Initial Insertion of Records
+ val dataGen = new HoodieTestDataGenerator()
+ val initialRecords =
recordsToStrings(dataGen.generateInserts(getInstantTime, 50, true)).asScala
+ val initialDf =
spark.read.json(spark.sparkContext.parallelize(initialRecords.toSeq, 2))
+ val hudiOpts = commonOpts ++ Map(TABLE_TYPE.key -> "MERGE_ON_READ",
HoodieWriteConfig.TBL_NAME.key -> tableName)
+ initialDf.write.format("hudi")
+.options(hudiOpts)
+.option(OPERATION.key, INSERT_OPERATION_OPT_VAL)
+.mode(SaveMode.Overwrite)
+.save(basePath)
+
+ // Step 2: Create table and secondary index on 'rider' column
+ spark.sql(s"CREATE TABLE $tableName USING hudi LOCATION '$basePath'")
+ spark.sql(s"create index idx_rider on $tableName using
secondary_index(rider)")
+
+ // Verify initial state of secondary index
+ val initialKeys = spark.sql(s"select _row_key from $tableName limit
5").collect().map(_.getString(0))
+ validateSecondaryIndex(basePath, tableName, initialKeys)
+
+ // Step 3: Perform Update Operations on Subset of Records
+ val records =
recordsToStrings(dataGen.generateUniqueUpdates(getInstantTime, 10,
HoodieTestDataGenerator.TRIP_FLATTENED_SCHEMA)).asScala
+ val df =
spark.read.json(spark.sparkContext.parallelize(records.toSeq, 2))
+ // Verify secondary index update fails
Review Comment:
added an update with secondary index disabled and ti goes through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
codope commented on code in PR #12280:
URL: https://github.com/apache/hudi/pull/12280#discussion_r1850678915
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestSecondaryIndex.scala:
##
@@ -284,6 +286,50 @@ class TestSecondaryIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Secondary Index With Overwrite and Delete Partition") {
+if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+Seq(
+ WriteOperationType.INSERT_OVERWRITE.value(),
+ WriteOperationType.INSERT_OVERWRITE_TABLE.value(),
+ WriteOperationType.DELETE_PARTITION.value()
+).foreach { operationType =>
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ // Step 1: Initial Insertion of Records
+ val dataGen = new HoodieTestDataGenerator()
+ val initialRecords =
recordsToStrings(dataGen.generateInserts(getInstantTime, 50, true)).asScala
+ val initialDf =
spark.read.json(spark.sparkContext.parallelize(initialRecords.toSeq, 2))
+ val hudiOpts = commonOpts ++ Map(TABLE_TYPE.key -> "MERGE_ON_READ",
HoodieWriteConfig.TBL_NAME.key -> tableName)
+ initialDf.write.format("hudi")
+.options(hudiOpts)
+.option(OPERATION.key, INSERT_OPERATION_OPT_VAL)
+.mode(SaveMode.Overwrite)
+.save(basePath)
+
+ // Step 2: Create table and secondary index on 'rider' column
+ spark.sql(s"CREATE TABLE $tableName USING hudi LOCATION '$basePath'")
+ spark.sql(s"create index idx_rider on $tableName using
secondary_index(rider)")
+
+ // Verify initial state of secondary index
+ val initialKeys = spark.sql(s"select _row_key from $tableName limit
5").collect().map(_.getString(0))
+ validateSecondaryIndex(basePath, tableName, initialKeys)
+
+ // Step 3: Perform Update Operations on Subset of Records
+ val records =
recordsToStrings(dataGen.generateUniqueUpdates(getInstantTime, 10,
HoodieTestDataGenerator.TRIP_FLATTENED_SCHEMA)).asScala
+ val df =
spark.read.json(spark.sparkContext.parallelize(records.toSeq, 2))
+ // Verify secondary index update fails
Review Comment:
Let's also check one more step after this verification i.e. disable
secondary index and validate that data table update goes through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2488689140 ## CI report: * c3a4cabc61d92019af39d6a23a90c8f8594f46d0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1774) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2488513069 ## CI report: * 73929570faf175fb25136b0f4f4f70a18b8bafa6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1770) * c3a4cabc61d92019af39d6a23a90c8f8594f46d0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1774) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2488507911 ## CI report: * 73929570faf175fb25136b0f4f4f70a18b8bafa6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1770) * c3a4cabc61d92019af39d6a23a90c8f8594f46d0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2488404361 ## CI report: * 73929570faf175fb25136b0f4f4f70a18b8bafa6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1770) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
lokeshj1703 commented on code in PR #12280:
URL: https://github.com/apache/hudi/pull/12280#discussion_r1850099690
##
hudi-common/src/main/java/org/apache/hudi/common/model/WriteOperationType.java:
##
@@ -123,6 +123,10 @@ public static boolean isOverwrite(WriteOperationType
operationType) {
return operationType == INSERT_OVERWRITE || operationType ==
INSERT_OVERWRITE_TABLE;
}
+ public boolean isOverwriteOrDeletePartition() {
Review Comment:
Addressed
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/BaseActionExecutor.java:
##
@@ -157,7 +157,7 @@ protected final void
writeTableMetadata(HoodieRestoreMetadata metadata) {
*/
protected final void dropIndexOnRestore() {
for (String partitionPath :
table.getMetaClient().getTableConfig().getMetadataPartitions()) {
- if (MetadataPartitionType.shouldDeletePartitionOnRestore(partitionPath))
{
+ if
(MetadataPartitionType.shouldDeleteMDTPartitionOnOverwriteOrRestore(partitionPath))
{
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
lokeshj1703 commented on code in PR #12280:
URL: https://github.com/apache/hudi/pull/12280#discussion_r1850099907
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/BaseActionExecutor.java:
##
@@ -157,7 +157,7 @@ protected final void
writeTableMetadata(HoodieRestoreMetadata metadata) {
*/
protected final void dropIndexOnRestore() {
for (String partitionPath :
table.getMetaClient().getTableConfig().getMetadataPartitions()) {
- if (MetadataPartitionType.shouldDeletePartitionOnRestore(partitionPath))
{
+ if
(MetadataPartitionType.shouldDeleteMDTPartitionOnOverwriteOrRestore(partitionPath))
{
Review Comment:
Addressed. We were not recreating the index, it was just some leftover code
from earlier attempts. Removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2488277525 ## CI report: * e91a071e369c6a77df692caff5812f06cb7ef165 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1726) * 73929570faf175fb25136b0f4f4f70a18b8bafa6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1770) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2488273131 ## CI report: * e91a071e369c6a77df692caff5812f06cb7ef165 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1726) * 73929570faf175fb25136b0f4f4f70a18b8bafa6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
nsivabalan commented on code in PR #12280:
URL: https://github.com/apache/hudi/pull/12280#discussion_r1847762115
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/BaseActionExecutor.java:
##
@@ -157,7 +157,7 @@ protected final void
writeTableMetadata(HoodieRestoreMetadata metadata) {
*/
protected final void dropIndexOnRestore() {
for (String partitionPath :
table.getMetaClient().getTableConfig().getMetadataPartitions()) {
- if (MetadataPartitionType.shouldDeletePartitionOnRestore(partitionPath))
{
+ if
(MetadataPartitionType.shouldDeleteMDTPartitionOnOverwriteOrRestore(partitionPath))
{
Review Comment:
lets not re-create the index automatically. its ok to throw exception and
suggest users to disable SI for the table of interest.
they can wait for 1.0.1 or something to get a fix for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
nsivabalan commented on code in PR #12280:
URL: https://github.com/apache/hudi/pull/12280#discussion_r1847759841
##
hudi-common/src/main/java/org/apache/hudi/common/model/WriteOperationType.java:
##
@@ -123,6 +123,10 @@ public static boolean isOverwrite(WriteOperationType
operationType) {
return operationType == INSERT_OVERWRITE || operationType ==
INSERT_OVERWRITE_TABLE;
}
+ public boolean isOverwriteOrDeletePartition() {
Review Comment:
isInsertOverwriteOrDeletePartition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2482762540 ## CI report: * 9d02bf49dcf9c3b9b6ebae57c6f646f4228e4b9f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1720) * 743a95b63ceb1f6eddb7d5c73c3e4a50947f562b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1721) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2483326246 ## CI report: * e91a071e369c6a77df692caff5812f06cb7ef165 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1726) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2482893074 ## CI report: * 743a95b63ceb1f6eddb7d5c73c3e4a50947f562b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1721) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2483097999 ## CI report: * 743a95b63ceb1f6eddb7d5c73c3e4a50947f562b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1721) * e91a071e369c6a77df692caff5812f06cb7ef165 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2483103115 ## CI report: * 743a95b63ceb1f6eddb7d5c73c3e4a50947f562b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1721) * e91a071e369c6a77df692caff5812f06cb7ef165 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1726) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2482645616 ## CI report: * 9d02bf49dcf9c3b9b6ebae57c6f646f4228e4b9f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1720) * 743a95b63ceb1f6eddb7d5c73c3e4a50947f562b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2482650577 ## CI report: * 9d02bf49dcf9c3b9b6ebae57c6f646f4228e4b9f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1720) * 743a95b63ceb1f6eddb7d5c73c3e4a50947f562b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2482605887 ## CI report: * 9d02bf49dcf9c3b9b6ebae57c6f646f4228e4b9f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1720) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported [hudi]
hudi-bot commented on PR #12280: URL: https://github.com/apache/hudi/pull/12280#issuecomment-2482593667 ## CI report: * 9d02bf49dcf9c3b9b6ebae57c6f646f4228e4b9f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
