JingsongLi commented on code in PR #11296:
URL: https://github.com/apache/paimon/pull/11296#discussion_r4642866696
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/execution/TruncatePaimonTableWithFilterExec.scala:
##########
@@ -36,37 +39,54 @@ case class TruncatePaimonTableWithFilterExec(
extends PaimonLeafV2CommandExec {
override def run(): Seq[InternalRow] = {
+ partitionPredicate match {
+ case Some(p) =>
+ table match {
+ case fileStoreTable: FileStoreTable =>
+ val matchedPartitions =
+
fileStoreTable.newSnapshotReader().withPartitionFilter(p).partitions()
+ if (matchedPartitions.isEmpty) {
+ withCommit(table)(_.commit(JCollections.emptyList()))
+ } else {
+ truncateMatchedPartitions(fileStoreTable, matchedPartitions)
+ }
+ case _ =>
+ throw new UnsupportedOperationException("Unsupported truncate
table")
+ }
+ case _ =>
+ withCommit(table)(_.truncateTable())
+ }
+ Nil
+ }
+
+ /**
+ * Removes the data of the matched partitions by their stored values: an
empty static overwrite. A
+ * spec rebuilt from a stored value does not always parse back to it, such
as a blank string,
+ * whose name is the default partition name.
+ */
+ private def truncateMatchedPartitions(
+ fileStoreTable: FileStoreTable,
+ partitions: JList[BinaryRow]): Unit = {
+ val target =
+ fileStoreTable.copy(Map(CoreOptions.DYNAMIC_PARTITION_OVERWRITE.key ->
"false").asJava)
+ val commit = target.newCommit(CoreOptions.createCommitUser(new
Options(target.options())))
+ try {
+ commit
+ .withOperation(Snapshot.Operation.TRUNCATE)
+ .withOverwriteStaticPartitions(partitions)
+ .commit(JCollections.emptyList[CommitMessage]())
Review Comment:
**[P1] Keep filtered DELETE scoped to its matching partitions.** On a native
table partitioned by STRING `dt`, I inserted `(1, '2020-01-01')` and `(2,
'2099-01-01')`, then enabled `partition.expiration-time='7 d'`,
`partition.expiration-check-interval='0 s'`,
`partition.timestamp-pattern='$dt'` and
`partition.timestamp-formatter='yyyy-MM-dd'` with ALTER TABLE. On Spark 3.5.9,
`DELETE FROM t WHERE dt = '2099-01-01'` left row 1 with the master
implementation and left the table empty on this head. The ordinary batch commit
runs table-wide partition expiration after publishing the DELETE, so a
partition outside the predicate disappears before the statement returns. Please
add a typed partition truncate path that delegates to the same overwrite
primitive as the original truncate and retains its maintenance semantics. Cover
an expired, nonmatching partition in the SQL test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]