Hi Ordencia, I see the recent investment in flink-s3-fs-native might lead to this thread discussion. To clarify, FLIP-555 work is part of the wider initiative to provide cloud filesystem integrations without Hadoop dependencies [1]. It does not propose any change to Flink’s HDFS support.
I believe two separate questions should be distinguished: 1. Which filesystem should be used for durable checkpoint storage? 2. Does Flink’s direction toward disaggregated state make object storage preferable for actively accessed state? Disaggregated state may naturally benefit from object-storage-like systems, but that does not imply that S3 is now generally recommended over HDFS for conventional checkpoint storage. Likewise, FLIP-555 aims to make the native implementation the recommended S3 plugin; it does not recommend S3 over other durable filesystems. Bests, Samrat [1] https://cwiki.apache.org/confluence/x/9gDuGQ On Wed, Aug 12, 2026 at 1:33 PM Gabor Somogyi <[email protected]> wrote: > Hi Ordencia, > > S3 is getting more trending as before and that's the reason why the > community is putting quite some efforts into it. > There are megatrends in open source, one of them is to break up with Hadoop > dependencies. A good example > here is that the new native S3 connector has zero Hadoop dependencies. It's > important to say it out that it doesn't > mean HDFS support is intended to be dropped in the near future since still > used heavily. > > All in all if you choose from these two and works for your use-case then > you've made good decision🙂 > Of course both of them has certain constraints and trade-offs which should > be considered on your side... > > BR, > G > > > On Tue, Aug 11, 2026 at 11:46 PM Ordencia Wu <[email protected]> > wrote: > > > Hi all, > > > > Just wanted to give this a quick bump, the main question I’m hoping to > > understand is whether the community expects Flink to remain storage > system > > neutral (especially between s3 and hdfs) for the use case of durable > > checkpoint and state storage? This question is inspired by the recent S3 > > native plugin and disaggregated state work since it’s so far unclear > > whether these roadmaps reflect a longer-term preference toward object > > storage for new deployments. > > > > Any pointers to relevant discussions or FLIPs would be helpful! > > > > Thank you, > > Ordencia > > > > On 2026/08/08 19:00:12 Ordencia Wu wrote: > > > Hi all! I am selecting the durable checkpoint store for a multi-tenant > > Flink platform. It appears recent introduction of `flink-s3-fs-native` > > brought significant improvements to checkpointing on S3, and FLIP-555 [1] > > describes a path towards making it the recommended S3 implementation. I > am > > weighing between S3 and HDFS as my options, and was wondering if the > recent > > improvements to Flink’s S3 integration and performance would change the > > community’s guidance for durable checkpoint storage in new deployments, > > particularly on Kubernetes. Any thoughts on the following would be very > > helpful: > > > > > > 1. Does the community expect S3/object storage to become the preferred > > durable checkpoint storage model for new deployments, or is the intent > for > > S3 and HDFS to remain equally first-class options? > > > 2. Are there planned FLIPs or architectural directions around > > checkpointing, recovery, or disaggregated state management that are > likely > > to favor S3/object-storage characteristics over HDFS > > > 3. If both are expected to remain equally supported, what factors would > > you recommend using to choose between remote HDFS and S3 for checkpoint > > storage? > > > > > > For context, the platform supports a range of different Flink > workloads: > > > 1. Jobs on the platform range from stateless to 100GB+ in state size > > > 2. SLOs vary greatly across tenants and the jobs could have different > > checkpoint intervals ranging from 10s to 10m > > > > > > Any thoughts or pointers to related discussions would be greatly > > appreciated! > > > > > > Best, > > > Ordencia > > > > > > ——————————————————— > > > [1] > > > https://cwiki.apache.org/confluence/spaces/FLINK/pages/396790457/FLIP-555+Flink+Native+S3+FileSystem > > > > > > > > > > > > >
