Hi Samrat, First couple of words about intentions: I think the the old connectors has its strengths and weaknesses. The whole point of the new native connector is to have a single solution which supports all of our needs. We're proceeding well on the road but there are still some things needs to be done.
I think there are 2 main aspects what can be considered: 1. User side: every user must have enough time to try the new connector out and best case migrate to it. This will be not lightning fast as there are heavyweight users with a lot and extremely complex jobs. We'll be here to support all of them on the road with suggestions/bugfixes/etc... 2. Development side: We try to do our best to keep alive all 3 connectors but there are physical constrains here. Up until now there were no 100+ developers who wanted to develop/maintain these connectors. Realistically I would say we can develop the new and shiny native connector and provide blocker bugfixes for the old but not infinitely. All in all I would say, first we should reach a stable runs in production at couple of big players state and then deprecate the old connectors. I've intentionally not written feature parity support because we're shooting for rock stable basic feature set which is easy to use. I'm expecting phasing out old connectors would be a relatively slow process where we can start separate discussions later. Happy to hear other voices. BR, G On Thu, Jun 18, 2026 at 10:54 PM <[email protected]> wrote: > Thanks Samrat for bringing this discussion up and for your work on the > native S3 filesystem. In terms of the native S3 filesystem, it is bridging > the gap that previously required flink-s3-fs-hadoop for sink/source and > flink-s3-fs-presto for checkpointing — having a single connector that > provides the RecoverableWriter for exactly-once sinks and also serves > checkpointing well, on AWS SDK v2, removes that long-standing "two > connectors per job" split. So I agree native (FLIP-555) is the right > long-term direction. > > What I'd advocate is continuing to support flink-s3-fs-hadoop while native > matures. > > Best, > Diljeet(DJ) Singh > > On 2026/06/15 13:50:08 Samrat Deb wrote: > > Hi, > > > > While following the ongoing work around adding AWS SDK v2 support to > > flink-s3-fs-hadoop patch [1], I realised that Flink's S3 filesystem > > landscape has evolved recently and now stands in a state where > > wider discussion is required to drive the direction forward. > > > > Today, Flink effectively has three S3 filesystem implementations: > > > > - flink-s3-fs-hadoop > > - flink-s3-fs-presto > > - native-s3-fs, introduced through FLIP-555[2] recently > > > > With native-s3-fs already providing AWS SDK v2 support by default. It > also > > supports FileSystem source/sink along with RecoverableWriter for > > checkpointing. As discussed in the thread[3], the goal is to deprecate > > flink-s3-fs-hadoop and flink-s3-fs-presto moving forward. > > > > I'm unclear on how the overall S3 story is expected to evolve. Should > the > > community expect to maintain multiple S3 implementations long term, each > > serving different use cases, or is there an eventual consolidation > strategy > > in mind? > > > > I am not proposing any specific direction here. I am mainly looking to > > understand the long-term vision so that ongoing efforts around S3 support > > can be evaluated in that broader context. > > > > > > [1] https://github.com/apache/flink/pull/27026 > > [2] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-555%3A+Flink+Native+S3+FileSystem > > [3] https://lists.apache.org/thread/2bllhqlbv0pz6t95tsjbszpm9bp9911c > > > > Best, > > Samrat > >
