[
https://issues.apache.org/jira/browse/FLINK-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15031496#comment-15031496
]
ASF GitHub Bot commented on FLINK-3084:
---------------------------------------
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/1423
[FLINK-3084] FsStateBackend backs up very small state directly with the
metadata
For the File State Backend (`FsStateBackend`), this change avoids the very
small files (few bytes) frequently created for small states, like Kafka
Offsets, or individual counters.
State whose size is below a certain size (512 bytes) is stored in the state
handle directly, rather than in files.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink small_files
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1423.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1423
----
commit 5e0ea62c32fb61eb5823457264f295fc40cbbaec
Author: Stephan Ewen <[email protected]>
Date: 2015-11-26T17:46:49Z
[FLINK-3084] [streaming] FsStateBackend backs up very small state directly
with the metadata.
----
> File State Backend should not write very small state into files
> ---------------------------------------------------------------
>
> Key: FLINK-3084
> URL: https://issues.apache.org/jira/browse/FLINK-3084
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 0.10.0
> Reporter: Stephan Ewen
> Assignee: Stephan Ewen
> Fix For: 1.0.0
>
>
> Currently, the {{FsStateBackend}} writes all state into files. Some state
> (like Kafka Offsets) is so small that it adds unnecessary overhead, and
> sometimes the checkpointed file handles are larger than the actual state.
> Small state (below a certain threshold, say 1 KB) should not be stored in
> files, but directly in the state handles.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
