RyanLin5967 opened a new issue, #20482:
URL: https://github.com/apache/hudi/issues/20482
### Bug Description
**What happened:**
Currently, `validateTimestampAsOf` refuses any instant older than the last
clean's `earliestCommitToRetain`, so time travel to a savepointed commit fails
even though the cleaner kept that commit's files. Before the clean the same
query returns the savepoint's rows.
**What you expected:**
`TIMESTAMP AS OF` the savepointed commit returns `[Row(id=1, v='b')]` after
the clean, as it does before it.
**Steps to reproduce:**
```python
from pyspark.sql import SparkSession
spark = (SparkSession.builder
.config("spark.jars", "hudi-spark3.5-bundle_2.12-1.2.1.jar")
.config("spark.serializer",
"org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.extensions",
"org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
.config("spark.sql.catalog.spark_catalog",
"org.apache.spark.sql.hudi.catalog.HoodieCatalog")
.getOrCreate())
spark.sql("CREATE TABLE t (id INT, v STRING) USING hudi TBLPROPERTIES
(primaryKey = 'id')")
spark.sql("INSERT INTO t VALUES (1, 'a')")
spark.sql("UPDATE t SET v = 'b' WHERE id = 1")
spark.sql("UPDATE t SET v = 'c' WHERE id = 1")
spark.sql("UPDATE t SET v = 'd' WHERE id = 1")
c1, c2, c3, c4 = sorted(r.commit_time for r in spark.sql("CALL
show_commits(table => 't')").collect())
spark.sql(f"CALL create_savepoint(table => 't', commit_time => '{c2}')")
print(spark.sql(f"SELECT id, v FROM t TIMESTAMP AS OF '{c2}'").collect())
# [Row(id=1, v='b')]
spark.sql("CALL run_clean(table => 't', clean_policy =>
'KEEP_LATEST_COMMITS', retain_commits => 1)")
print(sorted(r.base_instant for r in spark.sql("CALL show_fsview_all(table
=> 't')").collect()) == [c2, c3, c4])
# True, so the savepoint kept c2's file
spark.sql("REFRESH TABLE t")
print(spark.sql(f"SELECT id, v FROM t TIMESTAMP AS OF '{c2}'").collect())
# pyspark.errors.exceptions.captured.IllegalArgumentException: Cleaner
cleaned up the timestamp of interest. Please ensure sufficient commits are
retained with cleaner for Timestamp as of query to work
```
`spark.read.format("hudi").option("as.of.instant", c2)` fails the same way.
### Environment
**Hudi version:** 1.2.1 (`hudi-spark3.5-bundle_2.12`).
`TimelineUtils.validateTimestampAsOf` is the same on master.
**Query engine:** Spark 3.5.5 (pyspark, local mode) on Java 17.0.21, macOS
26.7.1 arm64
**Relevant configs:** none beyond the defaults; copy-on-write table
### Logs and Stack Trace
```
java.lang.IllegalArgumentException: Cleaner cleaned up the timestamp of
interest. Please ensure sufficient commits are retained with cleaner for
Timestamp as of query to work
at
org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
at
org.apache.hudi.common.table.timeline.TimelineUtils.validateTimestampAsOf(TimelineUtils.java:430)
at
org.apache.hudi.BaseHoodieTableFileIndex.loadFileSlicesForPartitions(BaseHoodieTableFileIndex.java:276)
at
org.apache.hudi.BaseHoodieTableFileIndex.ensurePreloadedPartitions(BaseHoodieTableFileIndex.java:267)
at
org.apache.hudi.BaseHoodieTableFileIndex.getInputFileSlices(BaseHoodieTableFileIndex.java:253)
at
org.apache.hudi.HoodieFileIndex.prunePartitionsAndGetFileSlices(HoodieFileIndex.scala:380)
at
org.apache.hudi.HoodieFileIndex.filterFileSlices(HoodieFileIndex.scala:257)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
