[
https://issues.apache.org/jira/browse/BEAM-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik resolved BEAM-4711.
-----------------------------
Resolution: Fixed
Fix Version/s: 2.8.0
> LocalFileSystem.delete doesn't support globbing
> -----------------------------------------------
>
> Key: BEAM-4711
> URL: https://issues.apache.org/jira/browse/BEAM-4711
> Project: Beam
> Issue Type: Task
> Components: sdk-py-core
> Affects Versions: 2.5.0
> Reporter: Ryan Williams
> Assignee: Ryan Williams
> Priority: Minor
> Fix For: 2.8.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> I attempted to run {{wordcount_it_test:WordCountIT.test_wordcount_it}}
> locally with {{DirectRunner}}:
> {code}
> python setup.py nosetests \
> --tests
> apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it \
> --test-pipeline-options="--output=foo"
> {code}
> It failed in [the {{delete_files}} cleanup
> command|https://github.com/apache/beam/blob/a58f1ffaafb0e2ebcc73a1c5abfb05a15ec6a84b/sdks/python/apache_beam/examples/wordcount_it_test.py#L64]:
> {code}
> root: WARNING: Retry with exponential backoff: waiting for 11.1454450937
> seconds before retrying delete_files because we caught exception:
> BeamIOError: Delete operation failed with exceptions
> {'foo/1530557644/results*': IOError(OSError(2, 'No such file or directory'),)}
> Traceback for above exception (most recent call last):
> File "/Users/ryan/c/beam/sdks/python/apache_beam/utils/retry.py", line 184,
> in wrapper
> return fun(*args, **kwargs)
> File "/Users/ryan/c/beam/sdks/python/apache_beam/testing/test_utils.py",
> line 136, in delete_files
> FileSystems.delete(file_paths)
> File "/Users/ryan/c/beam/sdks/python/apache_beam/io/filesystems.py", line
> 282, in delete
> return filesystem.delete(paths)
> File "/Users/ryan/c/beam/sdks/python/apache_beam/io/localfilesystem.py",
> line 304, in delete
> raise BeamIOError("Delete operation failed", exceptions)
> {code}
> The line:
> {code}
> self.addCleanup(delete_files, [output + '*'])
> {code}
> works as expected in GCS, and deletes a test's output-directory, but it fails
> in on the local-filesystem, which doesn't expand globs before attempting to
> delete paths.
> It would be good to make these consistent, presumably by adding glob-support
> to {{LocalFileSystem}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
