Hi everyone, Following up on our earlier Thread[1] regarding the architectural fragmentation of S3 support, I would like to formally present the progress on introducing a native S3 filesystem for Flink.
The current "dual-connector" ecosystem—split between flink-s3-fs-hadoop and flink-s3-fs-presto—has reached its technical limits. The Hadoop-based implementation introduces significant dependency bloat and persistent classpath conflicts, while the Presto-based connector lacks RecoverableWriter forcing users to manage multiple configurations for exactly-once sinks. To resolve this, I am proposing FLIP-555: Flink Native S3 FileSystem[2]. This implementation is built directly on the AWS SDK for Java v2, providing a unified, high-performance, and Hadoop-free solution for all S3 interactions. I have conducted benchmarking comparing the native implementation against the existing Presto-based filesystem. The initial results are highly motivating, with a visible performance gain. You can find the detailed performance analysis here[3] Following offline discussions with Piotr Nowojski and Gabor Somogyi, the POC and benchmarking results are good enough to validate that Native S3 FileSystem would be a valuable addition to Flink. With the addition of the Native S3 FileSystem, I have also discussed briefly the Deprecation Strategy to ensure operational stability in the FLIP. 1. Phase 1: Introduce flink-s3-fs-native as an optional plugin for community validation. 2. Phase 2: Promote the native connector to the recommended default once feature parity and stability are proven. 3. Phase 3: Formally deprecate the legacy Hadoop and Presto connectors in a future major release. Looking forward to your feedback and suggestions on the design and implementation details outlined in the FLIP. Cheers, Samrat [1] https://lists.apache.org/thread/2bllhqlbv0pz6t95tsjbszpm9bp9911c [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-555%3A+Flink+Native+S3+FileSystem [3] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620396
