Hi all, Just wanted to give this a quick bump, the main question I’m hoping to understand is whether the community expects Flink to remain storage system neutral (especially between s3 and hdfs) for the use case of durable checkpoint and state storage? This question is inspired by the recent S3 native plugin and disaggregated state work since it’s so far unclear whether these roadmaps reflect a longer-term preference toward object storage for new deployments.
Any pointers to relevant discussions or FLIPs would be helpful! Thank you, Ordencia On 2026/08/08 19:00:12 Ordencia Wu wrote: > Hi all! I am selecting the durable checkpoint store for a multi-tenant Flink > platform. It appears recent introduction of `flink-s3-fs-native` brought > significant improvements to checkpointing on S3, and FLIP-555 [1] describes a > path towards making it the recommended S3 implementation. I am weighing > between S3 and HDFS as my options, and was wondering if the recent > improvements to Flink’s S3 integration and performance would change the > community’s guidance for durable checkpoint storage in new deployments, > particularly on Kubernetes. Any thoughts on the following would be very > helpful: > > 1. Does the community expect S3/object storage to become the preferred > durable checkpoint storage model for new deployments, or is the intent for S3 > and HDFS to remain equally first-class options? > 2. Are there planned FLIPs or architectural directions around checkpointing, > recovery, or disaggregated state management that are likely to favor > S3/object-storage characteristics over HDFS > 3. If both are expected to remain equally supported, what factors would you > recommend using to choose between remote HDFS and S3 for checkpoint storage? > > For context, the platform supports a range of different Flink workloads: > 1. Jobs on the platform range from stateless to 100GB+ in state size > 2. SLOs vary greatly across tenants and the jobs could have different > checkpoint intervals ranging from 10s to 10m > > Any thoughts or pointers to related discussions would be greatly appreciated! > > Best, > Ordencia > > ——————————————————— > [1] > https://cwiki.apache.org/confluence/spaces/FLINK/pages/396790457/FLIP-555+Flink+Native+S3+FileSystem > > > >
