JingsongLi commented on PR #10964: URL: https://github.com/apache/paimon/pull/10964#issuecomment-6601385659
Could we keep `.done` cleanup inside `HiveCatalog.dropPartitions` instead of adding new public methods to `Catalog` and `PartitionModification`? After deleting the requested data partitions, HiveCatalog could derive the corresponding marker specs from those original specs and remove only their HMS entries, without truncating data or creating another snapshot. Normal and chain-table expiration would then use the ordinary partition-drop path without a second call for markers. This would keep the Hive marker convention within HiveCatalog and avoid expanding the public API for this fix. The cleanup needs to account for the shared HMS namespace: retain a marker if any branch still owns its underlying ordinary partition, or if any branch, including the current branch, contains a real partition whose literal spec equals the marker spec. These checks should use physical branch partitions without fallback reads and compare rendered partition specs, rather than converting synthetic values such as `20230101.done` to the partition type. Marker identity should come from deriving it from the original deletion request, not from treating every `.done` suffix as synthetic; an explicit DROP of a real STRING partition ending in `.done` must still delete its data with the usual branch protection for HMS metadata. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
