qiaoborui commented on PR #9833: URL: https://github.com/apache/seatunnel/pull/9833#issuecomment-3268746882
> > Can we provide a best practice to guide customers in configuration, such as recommended calculation formulas > > @zhangshenghang Based on local performance tests, increasing the partition count provides significant benefits when the number of tasks exceeds approximately 20,000. As a practical guideline, a partition count of around 1,000-2,000 tends to offer the best balance between reducing lock contention and minimizing overhead. It is recommended to start with this value and then adjust based on your cluster size and workload characteristics. Please note that these recommendations are based on my test results, and the optimal values may vary depending on your environment. Thank you for pointing this out. I've been keeping an eye on this issue recently as well. However, I previously reviewed these two PRs and noticed that the memory leaks addressed were related to `RunningJobStateIMap` and `pendingJobMasterMap`. I didn't observe any changes concerning `RunningJobMetrics`. Nevertheless, I'll give the latest PRs modifying the IMaps a try. Thank you very much for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
