[
https://issues.apache.org/jira/browse/FLINK-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323354#comment-17323354
]
Flink Jira Bot commented on FLINK-11695:
----------------------------------------
This issue is assigned but has not received an update in 7 days so it has been
labeled "stale-assigned". If you are still working on the issue, please give an
update and remove the label. If you are no longer working on the issue, please
unassign so someone else may work on it. In 7 days the issue will be
automatically unassigned.
> Make sharedStateDir could create sub-directories to avoid
> MaxDirectoryItemsExceededException
> --------------------------------------------------------------------------------------------
>
> Key: FLINK-11695
> URL: https://issues.apache.org/jira/browse/FLINK-11695
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Reporter: Yun Tang
> Assignee: Yun Tang
> Priority: Major
> Labels: stale-assigned
>
> We meet this annoying problem many times when the {{sharedStateDir}} in
> checkpoint path exceeded the directory items limit due to large checkpoint:
> {code:java}
> org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException:
> The directory item limit of xxx is exceeded: limit=1048576 items=1048576
> {code}
> Currently, our solution is to let {{FsCheckpointStorage}} could create
> sub-dirs when calling {{resolveCheckpointStorageLocation}}. The default value
> for the number of sub-dirs is zero, just keep backward compatibility as
> current situation. The created sub-dirs have the name as integer value of
> [{{0, num-of-sub-dirs}})
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
