JingsongLi commented on code in PR #10851:
URL: https://github.com/apache/paimon/pull/10851#discussion_r4456533665
##########
paimon-core/src/main/java/org/apache/paimon/operation/NormalPartitionExpire.java:
##########
@@ -161,14 +160,15 @@ private List<Map<String, String>> doExpire(
List<List<String>> expiredPartValues = new
ArrayList<>(partitionEntries.size());
for (PartitionEntry partition : partitionEntries) {
Object[] array = strategy.convertPartition(partition.partition());
- // DATE values are normalized for time extraction; use the stored
names for dropping.
+ // DATE and TIMESTAMP values are normalized for time extraction;
use the stored
+ // names for dropping.
Map<String, String> partitionSpec =
commit.pathFactory()
.partitionComputer()
.generatePartValues(partition.partition());
int index = 0;
for (String value : partitionSpec.values()) {
- if (array[index] instanceof LocalDate) {
+ if (array[index] != null &&
strategy.isNormalizedForTimeExtraction(index)) {
array[index] = value;
Review Comment:
[P1] Preserve the LTZ partition key when dropping legacy names
The legacy LTZ name is a UTC `Timestamp.toString()` spelling, but
`commit.dropPartitions` reparses the supplied string as local time. Replacing
the normalized value with this name therefore targets a different key. I
reproduced actual writes/expiry/readback with `TIMESTAMP_LTZ(0)`, legacy
partition names and `America/New_York`: write instants `2023-01-01T00:00Z` and
`2023-01-01T05:00Z`, then expire at local January 1 22:00 with one-day
retention. The returned selection correctly contains only legacy name
`2023-01-01T00:00` (the 27-hour-old partition), but readback leaves that old
row and removes the 05:00Z row, which is only 22 hours old. The legacy name was
reparsed as midnight New York, i.e. 05:00Z. The new normalization activates
this deletion path where the old default extractor skipped the ISO values.
Please make truncation round-trip the original typed LTZ key (or use a
legacy-aware conversion), keeping stored/HMS spelling separate where necessary,
and cover a negative-offse
t timezone with two such partitions. This is a logical deletion of the wrong
live partition, not a claim of immediate physical file erasure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
