JingsongLi commented on code in PR #9131:
URL: https://github.com/apache/paimon/pull/9131#discussion_r3794135013
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/source/CompactorSourceBuilder.java:
##########
@@ -100,9 +100,6 @@ public CompactorSourceBuilder
withPartitionIdleTime(@Nullable Duration partition
if (partitionPredicate != null) {
readBuilder.withPartitionFilter(partitionPredicate);
}
- if
(CoreOptions.fromMap(table.options()).manifestDeleteFileDropStats()) {
- readBuilder = readBuilder.dropStats();
Review Comment:
[P2] Could we keep dropping stats for the bounded compactor? In batch mode,
`CompactBucketsTable.BucketsRead` only checks the schema ID of each file and
then emits an empty file list; `SIZE_AWARE_BATCH` uses only `fileSize`. Without
`dropStats()`, bounded compaction retains full `valueStats` in the
JobManager-planned `DataSplit`s and serializes them to source subtasks (and
pending-split checkpoints, when enabled), only to discard them in the reader.
With the default `metadata.stats-mode=truncate(16)` and
`compaction.file-num-limit` allowing up to 100,000 files, this can materially
increase JobManager heap, serialization CPU, and network usage for wide tables.
A minimal fix would apply `readBuilder.dropStats()` only when `!isContinuous &&
manifestDeleteFileDropStats()`, plus add a batch-source regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
