xccui opened a new issue, #8940: URL: https://github.com/apache/hudi/issues/8940
**Describe the problem you faced** The compaction plan generation scheduler seems to be not working properly for our Flink Hudi writer job. We used the following settings and saw too frequent plan generation for a single table in JobManager logs. Flink checkpoint settings: ``` flinkcheckpoint.interval: 5 mins flinkcheckpoint.min.interval: 3 mins ``` Hudi compaction settings: ``` compaction.trigger.strategy: num_and_time compaction.delta_commits: 5 compaction.delta_seconds: 1200 ``` logs: ``` 2023-06-08 19:56:50 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - The delta commits >= 5 and elapsed_time >=1200s, trigger compaction scheduler. 2023-06-08 19:56:50 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Generating compaction plan for merge on read table s3a://path-to-table ... 2023-06-08 20:04:03 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - The delta commits >= 5 and elapsed_time >=1200s, trigger compaction scheduler. 2023-06-08 20:04:03 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Generating compaction plan for merge on read table s3a://path-to-table ... 2023-06-08 20:11:35 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - The delta commits >= 5 and elapsed_time >=1200s, trigger compaction scheduler. 2023-06-08 20:11:35 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Generating compaction plan for merge on read table s3a://path-to-table ... 2023-06-08 20:18:36 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - The delta commits >= 5 and elapsed_time >=1200s, trigger compaction scheduler. 2023-06-08 20:18:36 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Generating compaction plan for merge on read table s3a://path-to-table ... 2023-06-08 20:25:58 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - The delta commits >= 5 and elapsed_time >=1200s, trigger compaction scheduler. 2023-06-08 20:25:58 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Generating compaction plan for merge on read table s3a://path-to-table ``` I checked `ScheduleCompactionActionExecutor.getLatestDeltaCommitInfo()`. It checks time-escaped/delta-commits-number from the last commit. I wonder if we should replace "last commit" from "last commit request". **Environment Description** * Hudi version : 0.13.1 * Flink version : 0.16.1 * Hadoop version : 3.3.4 * Storage (HDFS/S3/GCS..) : S3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
