xloya commented on code in PR #5160:
URL: https://github.com/apache/gravitino/pull/5160#discussion_r1805712111
##########
clients/client-python/gravitino/filesystem/gvfs.py:
##########
@@ -27,6 +27,8 @@
from fsspec.implementations.arrow import ArrowFSWrapper
from fsspec.utils import infer_storage_options
from pyarrow.fs import HadoopFileSystem
+from pyarrow.fs import GcsFileSystem
Review Comment:
In the current implementation, that all storage Python lib dependencies will
be introduced into PyGVFS. Although there will be no conflicts in most cases,
it may be better for users to only load the underlying storage dependencies
they need. I wonder if there is a better way to introduce other FileSystem
dependencies on demand, could you take a time to research this?
##########
clients/client-python/gravitino/filesystem/gvfs.py:
##########
@@ -786,5 +801,22 @@ def _get_filesystem(self, actual_file_location: str):
finally:
write_lock.release()
+ def _get_gcs_filesystem(self):
+ # get All keys from the options that start with
'gravitino.bypass.gcs.' and remove the prefix
+ gcs_options = {
+ key[len(GVFSConfig.GVFS_FILESYSTEM_BY_PASS_GCS) :]: value
+ for key, value in self._options.items()
+ if key.startswith(GVFSConfig.GVFS_FILESYSTEM_BY_PASS_GCS)
+ }
+
+ # get 'service-account-key' from gcs_options, if the key is not found,
throw an exception
+ service_account_key_path = gcs_options.get("service-account-key-path")
Review Comment:
Please define this config in the `GVFSConfig`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
