[
https://issues.apache.org/jira/browse/FLINK-23093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen reassigned FLINK-23093:
------------------------------------
Assignee: Lijie Wang
> Limit number of I/O pool and Future threads in Mini Cluster
> -----------------------------------------------------------
>
> Key: FLINK-23093
> URL: https://issues.apache.org/jira/browse/FLINK-23093
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
> Affects Versions: 1.13.0, 1.12.4
> Reporter: Stephan Ewen
> Assignee: Lijie Wang
> Priority: Major
> Fix For: 1.14.0
>
>
> When running tests on CI via the minicluster, the mini cluster typically
> spawns 100s of I/O threads, both in the MiniCluster I/O pool and in the TM
> I/O pool.
> The standard rule for the maximum pool size is 4*num-cores, but the number of
> cores can be fairly large these days. Various Java versions also mess up core
> counting when running in containers (JVM container might have been given 2
> cores as resource limits, but the JVM counts the system as a whole, like
> 64/128 cores).
> This is both a nuisance for debugging, and a big waste of memory (each thread
> takes by default around 1MB when spawned, so the test JVM wastes 100s of MBs
> for nothing).
> I would suggest to set a default of 8 I/O threads for the Mini Cluster. The
> scaling-with-cores is important for proper TM/JM deployments, but not for the
> Mini Cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
