koochiswathiTR commented on issue #11371:
URL: https://github.com/apache/hudi/issues/11371#issuecomment-2157823177
@soumilshah1995 @ad1happy2go ,
We don`t know, how to reproduce this issue.
To mitigate the issue, We would like to delete the partition , below are the
options we tried. **But None worked.**
**Option 1:**
Delete all the IDs and delete the partitionName
val tniPath = "s3://s3_bucket_name/tmp/testhudi/document"
val data = spark.read.format("org.apache.hudi").load(tniPath)
val deleteKeysDF = data.filter($"partitionName" ===
"partition_name").select("ID")
deleteKeysDF.count()
val writeStatus =
deleteKeysDF.write.format("hudi").option("hoodie.table.name",
"document").option("hoodie.datasource.write.recordkey.field",
"ID").option("hoodie.datasource.write.table.name",
"document").option(DataSourceWriteOptions.OPERATION_OPT_KEY,
DataSourceWriteOptions.DELETE_OPERATION_OPT_VAL).option("hoodie.upsert.shuffle.parallelism",
"2").mode(SaveMode.Append).save(tniPath)
**Option 2:**
Delete the Partition Directory
val tniPath = "s3://s3_bucket_name/tmp/testhudi/document/"
val data1 = spark.read.format("org.apache.hudi").load(tniPath)
data1.write.format("hudi").option("hoodie.table.name",
"novusdoc").option("hoodie.datasource.write.recordkey.field",
"ID").option("hoodie.datasource.write.table.name",
"document").option("hoodie.datasource.write.operation",
"delete_partition").option("hoodie.datasource.write.partitions.to.delete",
"patition_name").option("hoodie.upsert.shuffle.parallelism",
"2").mode(SaveMode.Append).save(tniPath)
Both are creating commits and but compaction is also not happening but these
files are not getting deleted.
counts of these deleted partitions are also not decreased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
