[
https://issues.apache.org/jira/browse/BEAM-13250?focusedWorklogId=761383&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761383
]
ASF GitHub Bot logged work on BEAM-13250:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Apr/22 02:59
Start Date: 24/Apr/22 02:59
Worklog Time Spent: 10m
Work Description: BigJerBD opened a new pull request, #17455:
URL: https://github.com/apache/beam/pull/17455
Improvement to resolve : https://issues.apache.org/jira/browse/BEAM-13250
first commit is identical to https://github.com/apache/beam/pull/15977, but
since I believe this is relevant , I am open to put some effort to integrate a
proper solution
This change is relevant for testing purpose. Being able to change
`storage_client` allows use gcs emulator or other gcs configurations. (for
example:
https://github.com/fsouza/fake-gcs-server/issues/623#issuecomment-1107093547 )
This way it is possible at the very least to do :
```
class TestGCSFileSystem(GCSFileSystem):
GCS_PREFIX = 'gs-test://'
@classmethod
def scheme(cls):
return 'gs-test'
@staticmethod
def get_gcsio():
return gcsio.GcsIO(
storage_client=storage.StorageV1(
url="http://my_custom_url...";,
# Other custom configurations ...
)
)
```
Please note that this is not only usefull for testing purpose, it allows
more flexibility to configure the storage_client for the price of adding
another user defined filesystem scheme
Another alternative could to instead add an beam options to be able to
replace the url or other options. This might be better since it does not a new
function to the GcsFileSystem that could be used by sdk users
Thanks :)
R: @aaltay @ihji
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [x] [**Choose
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and
mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI.
Issue Time Tracking
-------------------
Worklog Id: (was: 761383)
Time Spent: 2h 10m (was: 2h)
> GCSFileSystem is not facilitating injection of the gcs url
> ----------------------------------------------------------
>
> Key: BEAM-13250
> URL: https://issues.apache.org/jira/browse/BEAM-13250
> Project: Beam
> Issue Type: Improvement
> Components: io-py-gcp
> Affects Versions: 2.33.0
> Reporter: Mathieu Leduc-Hamel
> Priority: P2
> Fix For: Not applicable
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Inside `apache_beam.io.gcp.gcsfilesystem.GCSFileSystem` is directly
> instantiating for every operations an instance of
> `apache_beam.io.gcp.gcsio.GcsIO` without allowing anyone to make use of the
> flexibility of instantiating with a custom client storage.
> As you can see inside `GcsIO` there's an optional parameter called
> `storage_client` and that would give us the possibility to overwrite the
> storage client by example depending of the run level and potentially make use
> of a Gcs emulator locally when testing the infrastructure.
> Right now the only way we have to overwrite the gcs url is by monkey patching
> the default storage which is not most clean way of doing it.
>
> Potential solution:
> If `GCSFileSystem` was transformed by example by adding a single class method
> to generate every one of thse `GcsIO`, that would give us the possibility to
> create a custom `GCSFileSystem` class, inheriting from it, without
> duplicating all of it's code and to inject our desired url depending of the
> run level.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
