Hi all! I am selecting the durable checkpoint store for a multi-tenant Flink platform. It appears recent introduction of `flink-s3-fs-native` brought significant improvements to checkpointing on S3, and FLIP-555 [1] describes a path towards making it the recommended S3 implementation. I am weighing between S3 and HDFS as my options, and was wondering if the recent improvements to Flink’s S3 integration and performance would change the community’s guidance for durable checkpoint storage in new deployments, particularly on Kubernetes. Any thoughts on the following would be very helpful:
1. Does the community expect S3/object storage to become the preferred durable checkpoint storage model for new deployments, or is the intent for S3 and HDFS to remain equally first-class options? 2. Are there planned FLIPs or architectural directions around checkpointing, recovery, or disaggregated state management that are likely to favor S3/object-storage characteristics over HDFS 3. If both are expected to remain equally supported, what factors would you recommend using to choose between remote HDFS and S3 for checkpoint storage? For context, the platform supports a range of different Flink workloads: 1. Jobs on the platform range from stateless to 100GB+ in state size 2. SLOs vary greatly across tenants and the jobs could have different checkpoint intervals ranging from 10s to 10m Any thoughts or pointers to related discussions would be greatly appreciated! Best, Ordencia ——————————————————— [1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/396790457/FLIP-555+Flink+Native+S3+FileSystem
