JingsongLi commented on code in PR #10851:
URL: https://github.com/apache/paimon/pull/10851#discussion_r4439168936
##########
paimon-core/src/main/java/org/apache/paimon/partition/PartitionExpireStrategy.java:
##########
@@ -74,13 +77,43 @@ public List<String> toPartitionValue(Object[] array) {
return list;
}
+ /**
+ * Whether {@link #convertPartition} renders the value at this index in a
form meant for time
+ * extraction that can differ from the stored partition name.
+ */
+ public boolean isNormalizedForTimeExtraction(int index) {
+ switch
(toObjectArrayConverter.rowType().getTypeAt(index).getTypeRoot()) {
+ case DATE:
+ case TIMESTAMP_WITHOUT_TIME_ZONE:
+ case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
+ return true;
+ default:
+ return false;
+ }
+ }
+
public Object[] convertPartition(BinaryRow partition) {
Object[] values = toObjectArrayConverter.convert(partition);
RowType partitionType = toObjectArrayConverter.rowType();
for (int index = 0; index < values.length; index++) {
- if (values[index] != null
- && partitionType.getTypeAt(index).getTypeRoot() ==
DataTypeRoot.DATE) {
- values[index] = LocalDate.ofEpochDay((Integer) values[index]);
+ if (values[index] == null) {
+ continue;
+ }
+ DataType type = partitionType.getTypeAt(index);
+ switch (type.getTypeRoot()) {
+ case DATE:
+ values[index] = LocalDate.ofEpochDay((Integer)
values[index]);
+ break;
+ case TIMESTAMP_WITHOUT_TIME_ZONE:
+ case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
+ // Timestamp#toString uses the ISO 'T' separator, which
the default
+ // partition time formatter cannot parse. Render it as the
SQL string.
+ values[index] =
+ DateTimeUtils.formatTimestamp(
Review Comment:
[P2] Preserve existing explicit timestamp formatter inputs
The unconditional SQL rendering changes the input of an existing configured
`partition.timestamp-formatter`. A `TIMESTAMP(0)` partition with
`partition.timestamp-pattern=$f0` and formatter `yyyy-MM-dd'T'HH:mm` previously
receives `Timestamp.toString()` values such as `2023-01-01T00:00`, so
expiration works. It now receives `2023-01-01 00:00:00`; the configured
extractor cannot parse it, and the expired partition is silently retained.
I reproduced actual writes, expiration and readback for January 1/January 5
partitions at a January 6 check with two-day retention. Both legacy and current
partition-name modes retain the old partition on this head; the identical cases
expire it successfully with the two production files restored to the PR
merge-base. Please keep explicit formatter inputs compatible, or adapt
timestamp extraction to the configured formatter, while retaining the stored
names for deletion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
