kerwin-zk opened a new pull request, #10964:
URL: https://github.com/apache/paimon/pull/10964

   ### Purpose
   
   Partition expiration on a Hive-backed table with an INT partition key 
deletes the normal partition, then fails with `NumberFormatException: For input 
string: "20230101.done"`. The cleanup introduced in #7507 sends synthetic 
`.done` partition specs through `Catalog.dropPartitions`, which also truncates 
Paimon data and converts the marker to the table's partition type. This happens 
even when no done marker exists in HMS.
   
   Add an explicit `dropPartitionsFromMetastore` operation and use it for 
expired done markers. Hive reuses its existing metadata removal logic, 
including partition key ordering and protection for partitions referenced by 
other branches, without truncating data or creating another snapshot. Delegate 
and caching catalogs forward the operation, and the caching catalog invalidates 
its partition listing. Ordinary partition deletion keeps its existing behavior, 
including deletion of real STRING partitions whose values end in `.done`.
   
   ### Tests
   
   - Before the fix, the new Hive regression fails with the same 
`NumberFormatException` both with and without existing done markers.
   - After the fix, all 161 tests in `PartitionExpireTest`, 
`CachingCatalogTest`, `DelegateCatalogTest`, and `HiveCatalogTest` pass, with 
no skips. Coverage includes marker cleanup, retention of unexpired INT 
partitions and their markers, a single data snapshot for expiration, 
metadata-only removal preserving table data, ordinary deletion of a real STRING 
`.done` partition, and delegate/cache behavior.
   - Existing test inputs and assertions are unchanged; the fake metastore in 
`PartitionExpireTest` implements the new metadata-only operation.
   
   Local validation (JDK 17):
   
   ```sh
   mvn -B -nsu -pl paimon-hive/paimon-hive-catalog -am -Pfast-build \
     -Dmaven.test.additionalClasspath=/path/to/hadoop-client-runtime-3.3.5.jar \
     -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false \
     
-Dtest=PartitionExpireTest,CachingCatalogTest,DelegateCatalogTest,HiveCatalogTest
 package
   ```
   
   The extra classpath entry supplies the Hadoop client runtime corresponding 
to the client API on the local test classpath; no dependency changes are 
included in this PR.
   
   Standard reactor packaging without `fast-build` also passes (`mvn -B -nsu 
-pl paimon-hive/paimon-hive-catalog -am -DskipTests package`), including 
Spotless and Checkstyle. `git diff --check` passes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to