Thanks Samrat for bringing this discussion up and for your work on the native S3 filesystem. In terms of the native S3 filesystem, it is bridging the gap that previously required flink-s3-fs-hadoop for sink/source and flink-s3-fs-presto for checkpointing — having a single connector that provides the RecoverableWriter for exactly-once sinks and also serves checkpointing well, on AWS SDK v2, removes that long-standing "two connectors per job" split. So I agree native (FLIP-555) is the right long-term direction.
What I'd advocate is continuing to support flink-s3-fs-hadoop while native matures. Best, Diljeet(DJ) Singh On 2026/06/15 13:50:08 Samrat Deb wrote: > Hi, > > While following the ongoing work around adding AWS SDK v2 support to > flink-s3-fs-hadoop patch [1], I realised that Flink's S3 filesystem > landscape has evolved recently and now stands in a state where > wider discussion is required to drive the direction forward. > > Today, Flink effectively has three S3 filesystem implementations: > > - flink-s3-fs-hadoop > - flink-s3-fs-presto > - native-s3-fs, introduced through FLIP-555[2] recently > > With native-s3-fs already providing AWS SDK v2 support by default. It also > supports FileSystem source/sink along with RecoverableWriter for > checkpointing. As discussed in the thread[3], the goal is to deprecate > flink-s3-fs-hadoop and flink-s3-fs-presto moving forward. > > I'm unclear on how the overall S3 story is expected to evolve. Should the > community expect to maintain multiple S3 implementations long term, each > serving different use cases, or is there an eventual consolidation strategy > in mind? > > I am not proposing any specific direction here. I am mainly looking to > understand the long-term vision so that ongoing efforts around S3 support > can be evaluated in that broader context. > > > [1] https://github.com/apache/flink/pull/27026 > [2] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-555%3A+Flink+Native+S3+FileSystem > [3] https://lists.apache.org/thread/2bllhqlbv0pz6t95tsjbszpm9bp9911c > > Best, > Samrat >
