nzw921rx commented on issue #12339: URL: https://github.com/apache/seatunnel/issues/12339#issuecomment-5757068796
Thank you for the update. 1. Regarding `checkpoint.scheduler-dispatch-thread-num` in `SharedCheckpointScheduler`, I agree that we should first determine whether this configuration is necessary based on evidence. After changing the implementation to `CompletableFuture.allOf(completableFutureArray).get()`, most of my concerns have been addressed. If the dispatcher does not perform any I/O or RPC operations, this configuration does not seem necessary. 2. Regarding the new scheduling model, we now have a complete benchmarking framework. I think we should establish a dedicated benchmark for the new coordinator to evaluate the performance of `SharedCheckpointScheduler`. This could be submitted as a separate PR. It would help us evaluate the efficiency of the new `SharedCheckpointScheduler` and establish a complete performance baseline for it. Currently, `org.apache.seatunnel.benchmark.CheckpointingTimeBenchmark#checkpointSingleInput` invokes `createPendingCheckpoint` through reflection, which bypasses the `CheckpointScheduler`. As a result, the scheduling and coordination efficiency of both the old and new implementations remains a blind spot in our benchmarks. I suggest adding this benchmark so that we can use concrete evidence to determine whether the new implementation is better or worse than the previous one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
