gemini-code-assist[bot] commented on code in PR #38526:
URL: https://github.com/apache/beam/pull/38526#discussion_r3260005597
##########
sdks/python/apache_beam/runners/interactive/cache_manager.py:
##########
@@ -286,13 +291,14 @@ def raw_source(self, *labels):
return self._reader_class(self._glob_path(*labels))
def cleanup(self):
- if self._cache_dir.startswith('gs://'):
- from apache_beam.io.gcp import gcsfilesystem
- from apache_beam.options.pipeline_options import PipelineOptions
- fs = gcsfilesystem.GCSFileSystem(PipelineOptions())
- fs.delete([self._cache_dir + '/full/'])
- elif filesystems.FileSystems.exists(self._cache_dir):
- filesystems.FileSystems.delete([self._cache_dir])
+ try:
+ if self._cache_dir.startswith('gs://'):
+ fs = gcsfilesystem.GCSFileSystem(PipelineOptions())
Review Comment:
Since the top-level import of `gcsfilesystem` should be removed to avoid
dependency issues for non-GCP users, it should be imported locally here within
the GCS-specific logic block, as it was in the original implementation.
```suggestion
from apache_beam.io.gcp import gcsfilesystem
fs = gcsfilesystem.GCSFileSystem(PipelineOptions())
```
##########
sdks/python/apache_beam/runners/interactive/cache_manager.py:
##########
@@ -29,9 +30,13 @@
from apache_beam.io import filesystems
from apache_beam.io import textio
from apache_beam.io import tfrecordio
+from apache_beam.io.gcp import gcsfilesystem
Review Comment:
Moving the `gcsfilesystem` import to the top level introduces a hard
dependency on the `google-cloud-storage` library (part of the `gcp` extra).
This will cause an `ImportError` for users who do not have GCP dependencies
installed, even if they are only using local file systems. This import should
remain local to the `cleanup` method to maintain the optional nature of the GCP
dependency in the Beam SDK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
