sjwiesman commented on a change in pull request #14932:
URL: https://github.com/apache/flink/pull/14932#discussion_r583088569
##########
File path: docs/content/docs/ops/state/checkpoints.md
##########
@@ -35,6 +35,64 @@ the same semantics as a failure-free execution.
See [Checkpointing]({{< ref
"docs/dev/datastream/fault-tolerance/checkpointing" >}}) for how to enable and
configure checkpoints for your program.
+## Checkpoint Storage
+
+When checkpointing is activated, such state is persisted upon checkpoints to
guard against data loss and recover consistently.
+Where the state is persisted upon checkpoints depends on the chosen
**Checkpoint Storage**.
+
+## Available State Backends
+
+Out of the box, Flink bundles these checkpoint storage types:
+
+ - *JobManagerCheckpointStorage*
+ - *FileSystemCheckpointStorage*
+
+{{< hint info >}}
+If a checkpoint directory is configured `FileSystemCheckpointStorage` will be
used, otherwise the system will use the `JobManagerCheckpointStorage`.
+{{< /hint >}}
+
+### The JobManagerCheckpointStorage
+
+The *JobManagerCheckpointStorage* stores checkpoint snapshots on in the
JobManager heap.
+
+holds data internally as objects on the Java heap. Key/value state and window
operators hold hash tables
+that store the values, triggers, etc.
+
+Upon checkpoints, this state backend will snapshot the state and send it as
part of the checkpoint acknowledgement messages to the
+JobManager, which stores it on its heap as well.
+
+The MemoryStateBackend can be configured to fail the checkpoint if it goes
over a certain size to avoid `OutOfMemoryError`'s on the JobManager. To set
this feature, users can instantiate a `JobManagerCheckpointStorage` with the
corresponding max size:
+
+```java
+new JobManagerCheckpointStorage(MAX_MEM_STATE_SIZE);
+```
+
+Limitations of the JobManagerCheckpointStorage:
+
+ - The size of each individual state is by default limited to 5 MB. This
value can be increased in the constructor of the JobManagerCheckpointStorage.
+ - Irrespective of the configured maximal state size, the state cannot be
larger than the akka frame size (see [Configuration]({{< ref
"docs/deployment/config" >}})).
+ - The aggregate state must fit into the JobManager memory.
+
+The JobManagerCheckpointStorage is encouraged for:
+
+ - Local development and debugging
+ - Jobs that do hold little state, such as jobs that consist only of
record-at-a-time functions (Map, FlatMap, Filter, ...). The Kafka Consumer
requires very little state.
Review comment:
If you set a checkpoint directory with JobManagerCheckpointStorage and
have HA enabled, neither is required, then it is highly available. It will add
some additional context to make this clear.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
