JingsongLi commented on code in PR #7715:
URL: https://github.com/apache/paimon/pull/7715#discussion_r3443119539
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/procedure/SparkOrphanFilesClean.scala:
##########
@@ -88,6 +88,18 @@ case class SparkOrphanFilesClean(
.toDS()
.cache()
+ if (usedManifestFiles.isEmpty) {
Review Comment:
Same issue here: this only checks whether any snapshot/manifest-list names
were collected, not whether the referenced data files were protected.
`collectWithoutDataFileWithManifestFlag` emits the manifest-list names before
reading their contents; if concurrent expiration removes those manifest-lists,
`retryReadingFiles(..., emptyList())` yields no manifest entries, but
`usedManifestFiles` is still non-empty. The job then builds an empty
`dataFiles` dataset and anti-joins candidates only against the manifest-list
names, so old data files can still be deleted. Please make the abort condition
cover the data-file collection failure/missing manifest-list case as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
