Patrick Lucas created FLINK-33694:
-------------------------------------
Summary: GCS filesystem does not respect gs.storage.root.url
config option
Key: FLINK-33694
URL: https://issues.apache.org/jira/browse/FLINK-33694
Project: Flink
Issue Type: Bug
Components: FileSystems
Affects Versions: 1.17.2, 1.18.0
Reporter: Patrick Lucas
The GCS FileSystem's RecoverableWriter implementation uses the GCS SDK directly
rather than going through Hadoop. While support has been added to configure
credentials correctly based on the standard Hadoop implementation
configuration, no other options are passed through to the underlying client.
Because this only affects the RecoverableWriter-related codepaths, it can
result in very surprising differing behavior whether the FileSystem is being
used as a source or a sink—while a {{{}gs://{}}}-URI FileSource may work fine,
a {{{}gs://{}}}-URI FileSink may not work at all.
We use [fake-gcs-server|https://github.com/fsouza/fake-gcs-server] in testing,
and so we override the Hadoop GCS FileSystem config option
{{{}gs.storage.root.url{}}}. However, because this option is not considered
when creating the GCS client for the RecoverableWriter codepath, in a FileSink
the GCS FileSystem attempts to write to the real GCS service rather than
fake-gcs-server. At the same time, a FileSource works as expected, reading from
fake-gcs-server.
The fix should be fairly straightforward, reading the {{gs.storage.root.url}}
config option from the Hadoop FileSystem config in
[{{GSFileSystemOptions}}|https://github.com/apache/flink/blob/release-1.18.0/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystemOptions.java#L30]
and, if set, passing it to {{storageOptionsBuilder}} in
[{{GSFileSystemFactory}}|https://github.com/apache/flink/blob/release-1.18.0/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystemFactory.java].
The only workaround for this is to build a custom flink-gs-fs-hadoop JAR with a
patch and use it as a plugin.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
