[
https://issues.apache.org/jira/browse/FLINK-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhilong Hong updated FLINK-23354:
---------------------------------
Description:
Currently a TaskExecutor uses BlobCache to cache the blobs transported from
JobManager. The caches are the local file stored on the TaskExecutor. The blob
cache will not be cleaned up until one hour after the related job is finished.
In FLINK-23218, we are going to distribute the cached ShuffleDescriptors via
blob. When large amount of failovers happen, there will be a lot of cache
stored on local disk. The blob cache will occupy large amount of disk space. In
extreme cases, the blob would blow up the disk space.
So we need to add a limit size for the ShuffleDescriptors stored in
PermanentBlobCache on TaskExecutor, as described in the comments of
FLINK-23218. The main idea is to add a size limit and and delete the blobs in
LRU order if the size limit is exceeded. Before a blob item is cached,
TaskExecutor will firstly check the overall size of cache. If the overall size
exceeds the limit, the blob will be deleted in LRU order until the limit is not
exceeded anymore. For the blob cache that is deleted, if it is used afterwards,
it will be downloaded from the HA or the blob server again.
The default value of the size limit for the ShuffleDescriptors in
PermanentBlobCache on TaskExecutor will be 100 MiB.
was:
Currently a TaskExecutor uses BlobCache to cache the blobs transported from
JobManager. The caches are the local file stored on the TaskExecutor. The blob
cache will not be cleaned up until one hour after the related job is finished.
At present, JobInformation and TaskInformation are transported via blob. If a
lot of jobs are submitted, the blob cache will occupy large amount of disk
space. In FLINK-23218, we are going to distribute the cached ShuffleDescriptors
via blob. When large amount of failovers happen, there will be a lot of cache
stored on local disk. In extreme cases, the blob would blow up the disk space.
So we need to add a limit size for the blob cache on TaskExecutor, as described
in the comments of FLINK-23218. The main idea is to add a size limit and and
delete blobs in LRU order if the size limit is exceeded. Before a blob item is
cached, TaskExecutor will firstly check the overall size of cache. If the
overall size exceeds the limit, the blob will be deleted in LRU order until the
limit is not exceeded anymore. For the blob cache that is deleted, if it is
used afterwards, it will be downloaded from the blob server again.
The default value of the size limit of the blob cache on TaskExecutor will be
10GiB.
> Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
> --------------------------------------------------------------------------
>
> Key: FLINK-23354
> URL: https://issues.apache.org/jira/browse/FLINK-23354
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
> Reporter: Zhilong Hong
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0
>
>
> Currently a TaskExecutor uses BlobCache to cache the blobs transported from
> JobManager. The caches are the local file stored on the TaskExecutor. The
> blob cache will not be cleaned up until one hour after the related job is
> finished. In FLINK-23218, we are going to distribute the cached
> ShuffleDescriptors via blob. When large amount of failovers happen, there
> will be a lot of cache stored on local disk. The blob cache will occupy large
> amount of disk space. In extreme cases, the blob would blow up the disk space.
> So we need to add a limit size for the ShuffleDescriptors stored in
> PermanentBlobCache on TaskExecutor, as described in the comments of
> FLINK-23218. The main idea is to add a size limit and and delete the blobs in
> LRU order if the size limit is exceeded. Before a blob item is cached,
> TaskExecutor will firstly check the overall size of cache. If the overall
> size exceeds the limit, the blob will be deleted in LRU order until the limit
> is not exceeded anymore. For the blob cache that is deleted, if it is used
> afterwards, it will be downloaded from the HA or the blob server again.
> The default value of the size limit for the ShuffleDescriptors in
> PermanentBlobCache on TaskExecutor will be 100 MiB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
