[
https://issues.apache.org/jira/browse/FLINK-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhilong Hong updated FLINK-23354:
---------------------------------
Description:
_This is the part 3 of the optimization related to task deployments. For more
details about the overall description and the part 1, please see FLINK-23005.
For more details about the part 2 please see FLINK-23218._
Currently a TaskExecutor uses BlobCache to cache the blobs transported from
JobManager. The caches are the local file stored on the TaskExecutor. The blob
cache will not be cleaned up until one hour after the related job is finished.
In FLINK-23218, we are going to distribute the cached ShuffleDescriptors via
blob. When large amount of failovers happen, there will be a lot of cache
stored on local disk. The blob cache will occupy large amount of disk space. In
extreme cases, the blob would blow up the disk space.
So we need to add a limit size for the ShuffleDescriptors stored in
PermanentBlobCache on TaskExecutor, as described in the comments of
FLINK-23218. The main idea is to add a size limit and and delete the blobs in
LRU order if the size limit is exceeded. Before a blob item is cached,
TaskExecutor will firstly check the overall size of cache. If the overall size
exceeds the limit, the blob will be deleted in LRU order until the limit is not
exceeded anymore. For the blob cache that is deleted, if it is used afterwards,
it will be downloaded from the HA or the blob server again.
The default value of the size limit for the ShuffleDescriptors in
PermanentBlobCache on TaskExecutor will be 100 MiB.
was:
Currently a TaskExecutor uses BlobCache to cache the blobs transported from
JobManager. The caches are the local file stored on the TaskExecutor. The blob
cache will not be cleaned up until one hour after the related job is finished.
In FLINK-23218, we are going to distribute the cached ShuffleDescriptors via
blob. When large amount of failovers happen, there will be a lot of cache
stored on local disk. The blob cache will occupy large amount of disk space. In
extreme cases, the blob would blow up the disk space.
So we need to add a limit size for the ShuffleDescriptors stored in
PermanentBlobCache on TaskExecutor, as described in the comments of
FLINK-23218. The main idea is to add a size limit and and delete the blobs in
LRU order if the size limit is exceeded. Before a blob item is cached,
TaskExecutor will firstly check the overall size of cache. If the overall size
exceeds the limit, the blob will be deleted in LRU order until the limit is not
exceeded anymore. For the blob cache that is deleted, if it is used afterwards,
it will be downloaded from the HA or the blob server again.
The default value of the size limit for the ShuffleDescriptors in
PermanentBlobCache on TaskExecutor will be 100 MiB.
> Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
> --------------------------------------------------------------------------
>
> Key: FLINK-23354
> URL: https://issues.apache.org/jira/browse/FLINK-23354
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Zhilong Hong
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0
>
>
> _This is the part 3 of the optimization related to task deployments. For more
> details about the overall description and the part 1, please see FLINK-23005.
> For more details about the part 2 please see FLINK-23218._
> Currently a TaskExecutor uses BlobCache to cache the blobs transported from
> JobManager. The caches are the local file stored on the TaskExecutor. The
> blob cache will not be cleaned up until one hour after the related job is
> finished. In FLINK-23218, we are going to distribute the cached
> ShuffleDescriptors via blob. When large amount of failovers happen, there
> will be a lot of cache stored on local disk. The blob cache will occupy large
> amount of disk space. In extreme cases, the blob would blow up the disk space.
> So we need to add a limit size for the ShuffleDescriptors stored in
> PermanentBlobCache on TaskExecutor, as described in the comments of
> FLINK-23218. The main idea is to add a size limit and and delete the blobs in
> LRU order if the size limit is exceeded. Before a blob item is cached,
> TaskExecutor will firstly check the overall size of cache. If the overall
> size exceeds the limit, the blob will be deleted in LRU order until the limit
> is not exceeded anymore. For the blob cache that is deleted, if it is used
> afterwards, it will be downloaded from the HA or the blob server again.
> The default value of the size limit for the ShuffleDescriptors in
> PermanentBlobCache on TaskExecutor will be 100 MiB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
