JingsongLi commented on code in PR #7537:
URL: https://github.com/apache/paimon/pull/7537#discussion_r3384875382
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/source/CompactorSourceBuilder.java:
##########
@@ -141,6 +142,26 @@ public DataStreamSource<RowData> build() {
});
dataStream = new DataStreamSource<>(filterStream);
}
+ CoreOptions coreOptions = table.coreOptions();
Review Comment:
This filter only removes expired partitions from the compaction source, but
the compactor sink still uses the table with `write-only=false`, so
`table.newCommit(...)` will create and run `PartitionExpire` during the compact
commit. As a result, a job that is supposed to skip expired partitions can
still delete those partitions as a side effect of compacting the remaining
active partitions. Could we either disable partition expiration for this
compaction commit path, or document that enabling this option may also expire
the skipped partitions during the commit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
