kerwin-zk opened a new pull request, #10964:
URL: https://github.com/apache/paimon/pull/10964
### Purpose
Partition expiration on a Hive-backed table with an INT partition key
deletes the normal partition, then fails with `NumberFormatException: For input
string: "20230101.done"`. The cleanup introduced in #7507 sends synthetic
`.done` partition specs through `Catalog.dropPartitions`, which also truncates
Paimon data and converts the marker to the table's partition type. This happens
even when no done marker exists in HMS.
Add an explicit `dropPartitionsFromMetastore` operation and use it for
expired done markers. Hive reuses its existing metadata removal logic,
including partition key ordering and protection for partitions referenced by
other branches, without truncating data or creating another snapshot. Delegate
and caching catalogs forward the operation, and the caching catalog invalidates
its partition listing. Ordinary partition deletion keeps its existing behavior,
including deletion of real STRING partitions whose values end in `.done`.
### Tests
- Before the fix, the new Hive regression fails with the same
`NumberFormatException` both with and without existing done markers.
- After the fix, all 161 tests in `PartitionExpireTest`,
`CachingCatalogTest`, `DelegateCatalogTest`, and `HiveCatalogTest` pass, with
no skips. Coverage includes marker cleanup, retention of unexpired INT
partitions and their markers, a single data snapshot for expiration,
metadata-only removal preserving table data, ordinary deletion of a real STRING
`.done` partition, and delegate/cache behavior.
- Existing test inputs and assertions are unchanged; the fake metastore in
`PartitionExpireTest` implements the new metadata-only operation.
Local validation (JDK 17):
```sh
mvn -B -nsu -pl paimon-hive/paimon-hive-catalog -am -Pfast-build \
-Dmaven.test.additionalClasspath=/path/to/hadoop-client-runtime-3.3.5.jar \
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false \
-Dtest=PartitionExpireTest,CachingCatalogTest,DelegateCatalogTest,HiveCatalogTest
package
```
The extra classpath entry supplies the Hadoop client runtime corresponding
to the client API on the local test classpath; no dependency changes are
included in this PR.
Standard reactor packaging without `fast-build` also passes (`mvn -B -nsu
-pl paimon-hive/paimon-hive-catalog -am -DskipTests package`), including
Spotless and Checkstyle. `git diff --check` passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]