[
https://issues.apache.org/jira/browse/FLINK-9297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jelmer Kuperus updated FLINK-9297:
----------------------------------
Description:
With flink 1.4.0 cleaning of checkpoints is handled on the job manager, the
SharedStateRegistry uses the job manager's [io
executor|https://github.com/apache/flink/blob/e5ed2fbc4a31808bea8be89b371bbe4269288003/flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala#L1999]
as the executor to schedule disposals of checkpoint state on. This executor
has a fixed size equal to the number of cpu cores. When a lot of small files
are created by many jobs and a slow filesystem is used this may not be enough.
It would be good to make this setting configurable. Initializing an executor
service with the number of cpu's on the system makes sense for cpu intensive
tasks less so for io based workloads
was:
With flink 1.4.0 cleaning of checkpoints is handled on the job manager, the
SharedStateRegistry uses the job manager's [io
executor|https://github.com/apache/flink/blob/e5ed2fbc4a31808bea8be89b371bbe4269288003/flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala#L1999]
as the executor to schedule disposals of checkpoint state on. This executor
has a fixed size equal to the number of cpu cores. When a lot of small files
are created this may not be enough.
It would be good to make this setting configurable. Initializing an executor
service with the number of cpu's on the system makes sense for cpu intensive
tasks less so for io based workloads
> Make size of JobManager's ioExecutor configurable
> -------------------------------------------------
>
> Key: FLINK-9297
> URL: https://issues.apache.org/jira/browse/FLINK-9297
> Project: Flink
> Issue Type: Improvement
> Components: JobManager
> Affects Versions: 1.4.0, 1.4.1, 1.4.2
> Reporter: Jelmer Kuperus
> Priority: Major
>
> With flink 1.4.0 cleaning of checkpoints is handled on the job manager, the
> SharedStateRegistry uses the job manager's [io
> executor|https://github.com/apache/flink/blob/e5ed2fbc4a31808bea8be89b371bbe4269288003/flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala#L1999]
> as the executor to schedule disposals of checkpoint state on. This
> executor has a fixed size equal to the number of cpu cores. When a lot of
> small files are created by many jobs and a slow filesystem is used this may
> not be enough.
> It would be good to make this setting configurable. Initializing an executor
> service with the number of cpu's on the system makes sense for cpu intensive
> tasks less so for io based workloads
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
