[
https://issues.apache.org/jira/browse/FLINK-34984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zakelly Lan updated FLINK-34984:
--------------------------------
Description:
The past decade has witnessed a dramatic shift in Flink's deployment mode,
workload patterns, and hardware improvements. We've moved from the map-reduce
era where workers are computation-storage tightly coupled nodes to a
cloud-native world where containerized deployments on Kubernetes become
standard. To enable Flink's Cloud-Native future, we introduce Disaggregated
State Storage and Management that uses DFS as primary storage in Flink 2.0, as
promised in the Flink 2.0 Roadmap.
Detailed design and story:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855
Also sub-FLIPs:
- Asynchronous State APIs
([FLIP-424|https://cwiki.apache.org/confluence/x/SYp3EQ]): Introduce new APIs
for asynchronous state access.
- Asynchronous Execution Model
([FLIP-425|https://cwiki.apache.org/confluence/x/S4p3EQ]): Implement a
non-blocking execution model leveraging the asynchronous APIs introduced in
FLIP-424.
- Grouping Remote State Access
([FLIP-426|https://cwiki.apache.org/confluence/x/TYp3EQ]): Enable retrieval of
remote state data in batches to avoid unnecessary round-trip costs for remote
access.
- Disaggregated State Store
([FLIP-427|https://cwiki.apache.org/confluence/x/T4p3EQ]): Introduce the
initial version of the ForSt disaggregated state store.
- Fault Tolerance/Rescale Integration
([FLIP-428|https://cwiki.apache.org/confluence/x/UYp3EQ]): Integrate
checkpointing mechanisms with the disaggregated state store for fault tolerance
and fast rescaling.
was:The past decade has witnessed a dramatic shift in Flink's deployment
mode, workload patterns, and hardware improvements. We've moved from the
map-reduce era where workers are computation-storage tightly coupled nodes to a
cloud-native world where containerized deployments on Kubernetes become
standard. To enable Flink's Cloud-Native future, we introduce Disaggregated
State Storage and Management that uses DFS as primary storage in Flink 2.0, as
promised in the Flink 2.0 Roadmap.
> Disaggregated State Storage and Management (Umbrella FLIP)
> ----------------------------------------------------------
>
> Key: FLINK-34984
> URL: https://issues.apache.org/jira/browse/FLINK-34984
> Project: Flink
> Issue Type: New Feature
> Components: API / Core, API / DataStream, Runtime / Checkpointing,
> Runtime / State Backends
> Reporter: Yuan Mei
> Priority: Major
>
> The past decade has witnessed a dramatic shift in Flink's deployment mode,
> workload patterns, and hardware improvements. We've moved from the map-reduce
> era where workers are computation-storage tightly coupled nodes to a
> cloud-native world where containerized deployments on Kubernetes become
> standard. To enable Flink's Cloud-Native future, we introduce Disaggregated
> State Storage and Management that uses DFS as primary storage in Flink 2.0,
> as promised in the Flink 2.0 Roadmap.
> Detailed design and story:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855
> Also sub-FLIPs:
> - Asynchronous State APIs
> ([FLIP-424|https://cwiki.apache.org/confluence/x/SYp3EQ]): Introduce new APIs
> for asynchronous state access.
> - Asynchronous Execution Model
> ([FLIP-425|https://cwiki.apache.org/confluence/x/S4p3EQ]): Implement a
> non-blocking execution model leveraging the asynchronous APIs introduced in
> FLIP-424.
> - Grouping Remote State Access
> ([FLIP-426|https://cwiki.apache.org/confluence/x/TYp3EQ]): Enable retrieval
> of remote state data in batches to avoid unnecessary round-trip costs for
> remote access.
> - Disaggregated State Store
> ([FLIP-427|https://cwiki.apache.org/confluence/x/T4p3EQ]): Introduce the
> initial version of the ForSt disaggregated state store.
> - Fault Tolerance/Rescale Integration
> ([FLIP-428|https://cwiki.apache.org/confluence/x/UYp3EQ]): Integrate
> checkpointing mechanisms with the disaggregated state store for fault
> tolerance and fast rescaling.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
