Ryan Williams created BEAM-4711:
-----------------------------------
Summary: LocalFileSystem.delete doesn't support globbing
Key: BEAM-4711
URL: https://issues.apache.org/jira/browse/BEAM-4711
Project: Beam
Issue Type: Task
Components: sdk-py-core
Affects Versions: 2.5.0
Reporter: Ryan Williams
Assignee: Ryan Williams
I attempted to run {{wordcount_it_test:WordCountIT.test_wordcount_it}} locally
with {{DirectRunner}}:
{code}
python setup.py nosetests \
--tests apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it \
--test-pipeline-options="--output=foo"
{code}
It failed in [the {{delete_files}} cleanup
command|https://github.com/apache/beam/blob/a58f1ffaafb0e2ebcc73a1c5abfb05a15ec6a84b/sdks/python/apache_beam/examples/wordcount_it_test.py#L64]:
{code}
root: WARNING: Retry with exponential backoff: waiting for 11.1454450937
seconds before retrying delete_files because we caught exception: BeamIOError:
Delete operation failed with exceptions {'foo/1530557644/results*':
IOError(OSError(2, 'No such file or directory'),)}
Traceback for above exception (most recent call last):
File "/Users/ryan/c/beam/sdks/python/apache_beam/utils/retry.py", line 184,
in wrapper
return fun(*args, **kwargs)
File "/Users/ryan/c/beam/sdks/python/apache_beam/testing/test_utils.py", line
136, in delete_files
FileSystems.delete(file_paths)
File "/Users/ryan/c/beam/sdks/python/apache_beam/io/filesystems.py", line
282, in delete
return filesystem.delete(paths)
File "/Users/ryan/c/beam/sdks/python/apache_beam/io/localfilesystem.py", line
304, in delete
raise BeamIOError("Delete operation failed", exceptions)
{code}
The line:
{code}
self.addCleanup(delete_files, [output + '*'])
{code}
works as expected in GCS, and deletes a test's output-directory, but it fails
in on the local-filesystem, which doesn't expand globs before attempting to
delete paths.
It would be good to make these consistent, presumably by adding glob-support to
{{LocalFileSystem}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
