[
https://issues.apache.org/jira/browse/BEAM-13250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mathieu Leduc-Hamel updated BEAM-13250:
---------------------------------------
Description:
Inside `apache_beam.io.gcp.gcsfilesystem.GCSFileSystem` is directly
instantiating for every operations an instance of
`apache_beam.io.gcp.gcsio.GcsIO` without allowing anyone to make use of the
flexibility of instantiating with a custom client storage.
As you can see inside `GcsIO` there's an optional parameter called
`storage_client` and that would give us the possibility to overwrite the
storage client by example depending of the run level and potentially make use
of a Gcs emulator locally when testing the infrastructure.
Right now the only way we have to overwrite the gcs url is by monkey patching
the default storage which is not most clean way of doing it.
Potential solution:
If `GCSFileSystem` was transformed by example by adding a single class method
to generate every one of thse `GcsIO`, that would give us the possibility to
create a custom `GCSFileSystem` class, inheriting from it, without duplicating
all of it's code and to inject our desired url depending of the run level.
was:
Inside `apache_beam.io.gcp.gcsfilesystem.GCSFileSystem` is directly
instantiating for every operations an instance of
`apache_beam.io.gcp.gcsio.GcsIO` without allowing anyone to make use of the
flexibility of instantiating with a custom client storage.
As you can see inside `GcsIO` there's an optional parameter called
`storage_client` and that would give us the possibility to overwrite the
storage client by example depending of the run level and potentially make use
of a Gcs emulator locally when testing the infrastructure.
Right now the only way we have to overwrite the gcs url is by monkey patching
the default storage which is not most clean way of doing it.
Potential solution:
If `GCSFileSystem` was transformed by example by adding a single class method
to generate every one of the `GcsIO` that would give us the possibility to
create a custom `GCSFileSystem` class, inheriting from it, without duplicating
all of it's code and to inject our desired url depending of the run level.
> GCSFileSystem is not facilitating injection of the gcs url
> ----------------------------------------------------------
>
> Key: BEAM-13250
> URL: https://issues.apache.org/jira/browse/BEAM-13250
> Project: Beam
> Issue Type: Improvement
> Components: io-py-gcp
> Affects Versions: 2.33.0
> Reporter: Mathieu Leduc-Hamel
> Priority: P2
> Fix For: Not applicable
>
>
> Inside `apache_beam.io.gcp.gcsfilesystem.GCSFileSystem` is directly
> instantiating for every operations an instance of
> `apache_beam.io.gcp.gcsio.GcsIO` without allowing anyone to make use of the
> flexibility of instantiating with a custom client storage.
> As you can see inside `GcsIO` there's an optional parameter called
> `storage_client` and that would give us the possibility to overwrite the
> storage client by example depending of the run level and potentially make use
> of a Gcs emulator locally when testing the infrastructure.
> Right now the only way we have to overwrite the gcs url is by monkey patching
> the default storage which is not most clean way of doing it.
>
> Potential solution:
> If `GCSFileSystem` was transformed by example by adding a single class method
> to generate every one of thse `GcsIO`, that would give us the possibility to
> create a custom `GCSFileSystem` class, inheriting from it, without
> duplicating all of it's code and to inject our desired url depending of the
> run level.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
