kylebarron opened a new pull request, #322: URL: https://github.com/apache/arrow-rs-object-store/pull/322
# Which issue does this PR close? Closes https://github.com/apache/arrow-rs-object-store/issues/302. # Rationale for this change # What changes are included in this PR? - Add `skip_signature` config parameter - Add `with_skip_signature` to `GoogleCloudStorageBuilder` - Add `CredentialExt` in `src/gcp/credential.rs`, similar to what already exists for AWS in `src/aws/credential.rs`. This allows for easily calling `with_bearer_auth` in the request builder chain without having to do an `if/else` to handle `credentials` being `None` there. **Questions** - How should I test this? I looked for other existing tests for `skip_signature` and it looks like the one existing one for AWS is ignored: https://github.com/apache/arrow-rs-object-store/blob/e975177814778b711e11b048bb5c703d2b893427/src/aws/mod.rs#L690-L714 I was able to successfully test this in the obstore Python bindings (https://github.com/developmentseed/obstore/pull/404) on a GCP open data bucket: ```py store = GCSStore("gcp-public-data-arco-era5", skip_signature=True) ``` ```py next(store.list()) # [{'path': 'ar/1959-2022-1h-240x121_equiangular_with_poles_conservative.zarr/.zattrs', # 'last_modified': datetime.datetime(2023, 9, 14, 7, 6, 30, 187000, tzinfo=datetime.timezone.utc), # 'size': 2, # 'e_tag': '"99914b932bd37a50b983c5e7c90ae93b"', # 'version': None}, # {'path': 'ar/1959-2022-1h-240x121_equiangular_with_poles_conservative.zarr/.zgroup', # 'last_modified': datetime.datetime(2023, 9, 14, 7, 6, 28, 987000, tzinfo=datetime.timezone.utc), # 'size': 24, # 'e_tag': '"e20297935e73dd0154104d4ea53040ab"', # 'version': None}, # ... orjson.loads( memoryview( store.get( "ar/1959-2022-1h-240x121_equiangular_with_poles_conservative.zarr/.zmetadata", ).bytes() ), ) # {'metadata': {'.zattrs': {}, # '.zgroup': {'zarr_format': 2}, # '10m_u_component_of_wind/.zarray': {'chunks': [8, 240, 121], # 'compressor': {'blocksize': 0, # 'clevel': 5, # 'cname': 'lz4', # 'id': 'blosc', # 'shuffle': 1}, # 'dtype': '<f4', # 'fill_value': 'NaN', # 'filters': None, # 'order': 'C', # 'shape': [552264, 240, 121], # 'zarr_format': 2}, # '10m_u_component_of_wind/.zattrs': {'_ARRAY_DIMENSIONS': ['time', # 'longitude', # 'latitude'], # 'long_name': '10 metre U wind component', # 'short_name': 'u10', # 'units': 'm s**-1'}, # ... ``` (cc @paraseba @tomnicholas @maxrjones) # Are there any user-facing changes? Adds `skip_signature`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
