Hi All, Following the earlier discussion about a Hadoop-independent GCS filesystem [1], I would like to start the formal discussion for FLIP-603: GCS SDK-native filesystem [2]. FLIP-603 proposes a new flink-gcs-fs-native plugin that implements the gs:// filesystem directly using the Google Cloud Storage Java SDK, without depending on Hadoop or gcs-connector.
The existing flink-gs-fs-hadoop plugin is already partly native: ordinary filesystem operations are delegated to GoogleHadoopFileSystem, while its RecoverableWriter uses the GCS SDK and server-side compose directly. The proposal replaces the Hadoop-backed operations while retaining the existing compose-based recovery design. The initial proposal deliberately keeps the following outside its scope: - Deprecating flink-gs-fs-hadoop before the native implementation reaches feature parity and production maturity. - Client-side encryption, which belongs to the shared SDK-native filesystem work. - PathsCopyingFileSystem support. - GCS Hierarchical Namespace acceleration; standard copy-and-delete rename remains the compatibility baseline. - A cloud-generic object-storage operations interface. Looking forward to feedback an suggestion on the following: 1. Should we retain the existing gs.* configuration namespace for migration compatibility, or use a separate namespace for native-only options? 2. Should HNS-enabled buckets and atomic folder rename be supported in the first version or added later as an optimisation? 3. Are there existing gcs-connector directory, authentication, or endpoint behaviours that must be included in the initial compatibility target? 4. Is preserving the current GCS recoverable-state format sufficient, including the requirement to retain temporary-bucket and entropy settings during migration? 5. Should GCS bulk-copy support remain a follow-up, or is it required for initial feature parity? Bests, Samrat [1] https://lists.apache.org/thread/lhvkzjrwh7jqn1oj2m4g7dlhgggfrtrf [2] https://cwiki.apache.org/confluence/spaces/FLINK/pages/444334267/FLIP-603+FileSystems+GCS+SDK-native+filesystem [3] https://cwiki.apache.org/confluence/spaces/FLINK/pages/435028214/FLIP-597+Filesystems+Hadoop-less+Flink+filesystems
