[
https://issues.apache.org/jira/browse/FLINK-34984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zakelly Lan reassigned FLINK-34984:
-----------------------------------
Assignee: Zakelly Lan
> FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)
> --------------------------------------------------------------------
>
> Key: FLINK-34984
> URL: https://issues.apache.org/jira/browse/FLINK-34984
> Project: Flink
> Issue Type: New Feature
> Components: API / Core, API / DataStream, Runtime / Checkpointing,
> Runtime / State Backends
> Reporter: Yuan Mei
> Assignee: Zakelly Lan
> Priority: Major
>
> The past decade has witnessed a dramatic shift in Flink's deployment mode,
> workload patterns, and hardware improvements. We've moved from the map-reduce
> era where workers are computation-storage tightly coupled nodes to a
> cloud-native world where containerized deployments on Kubernetes become
> standard. To enable Flink's Cloud-Native future, we introduce Disaggregated
> State Storage and Management that uses DFS as primary storage in Flink 2.0
> This new architecture is aimed to solve the following challenges brought in
> the cloud-native era for Flink.
> 1. Local Disk Constraints in containerization
> 2. Spiky Resource Usage caused by compaction in the current state model
> 3. Fast Rescaling for jobs with large states (hundreds of Terabytes)
> 4. Light and Fast Checkpoint in a native way
>
> Design Details can be found in
> [FLIP-423|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855]
> Proposed changes can be found here:
> * [Asynchronous State APIs (FLIP-424)
> |https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-AsynchronousStateAPIs(FLIP-424)]
> * [Non-blocking Asynchronous Execution Model: Parallel I/O
> (FLIP-425)|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-Non-blockingAsynchronousExecutionModel:ParallelI/O(FLIP-425)]
> * [Batching for Network I/O: Beyond Parallel I/O
> (FLIP-426)|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-BatchingforNetworkI/O:BeyondParallelI/O(FLIP-426)]
> * [Disaggregated State Store: ForSt
> (FLIP-427)|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-DisaggregatedStateStore:ForSt(FLIP-427)]
> * [Faster Checkpoint/Restore/Rescale: Leverage Shared DFS
> (FLIP-428)|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-FasterCheckpoint/Restore/Rescale:LeverageSharedDFS(FLIP-428)]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
