[
https://issues.apache.org/jira/browse/BEAM-4711?focusedWorklogId=127624&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127624
]
ASF GitHub Bot logged work on BEAM-4711:
----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Jul/18 01:40
Start Date: 26/Jul/18 01:40
Worklog Time Spent: 10m
Work Description: ryan-williams commented on issue #5863: [BEAM-4711] fix
globbing in LocalFileSystem.delete
URL: https://github.com/apache/beam/pull/5863#issuecomment-407948787
I never pinged for review here; @lukecwik do you care to take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 127624)
Time Spent: 20m (was: 10m)
> LocalFileSystem.delete doesn't support globbing
> -----------------------------------------------
>
> Key: BEAM-4711
> URL: https://issues.apache.org/jira/browse/BEAM-4711
> Project: Beam
> Issue Type: Task
> Components: sdk-py-core
> Affects Versions: 2.5.0
> Reporter: Ryan Williams
> Assignee: Ryan Williams
> Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I attempted to run {{wordcount_it_test:WordCountIT.test_wordcount_it}}
> locally with {{DirectRunner}}:
> {code}
> python setup.py nosetests \
> --tests
> apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it \
> --test-pipeline-options="--output=foo"
> {code}
> It failed in [the {{delete_files}} cleanup
> command|https://github.com/apache/beam/blob/a58f1ffaafb0e2ebcc73a1c5abfb05a15ec6a84b/sdks/python/apache_beam/examples/wordcount_it_test.py#L64]:
> {code}
> root: WARNING: Retry with exponential backoff: waiting for 11.1454450937
> seconds before retrying delete_files because we caught exception:
> BeamIOError: Delete operation failed with exceptions
> {'foo/1530557644/results*': IOError(OSError(2, 'No such file or directory'),)}
> Traceback for above exception (most recent call last):
> File "/Users/ryan/c/beam/sdks/python/apache_beam/utils/retry.py", line 184,
> in wrapper
> return fun(*args, **kwargs)
> File "/Users/ryan/c/beam/sdks/python/apache_beam/testing/test_utils.py",
> line 136, in delete_files
> FileSystems.delete(file_paths)
> File "/Users/ryan/c/beam/sdks/python/apache_beam/io/filesystems.py", line
> 282, in delete
> return filesystem.delete(paths)
> File "/Users/ryan/c/beam/sdks/python/apache_beam/io/localfilesystem.py",
> line 304, in delete
> raise BeamIOError("Delete operation failed", exceptions)
> {code}
> The line:
> {code}
> self.addCleanup(delete_files, [output + '*'])
> {code}
> works as expected in GCS, and deletes a test's output-directory, but it fails
> in on the local-filesystem, which doesn't expand globs before attempting to
> delete paths.
> It would be good to make these consistent, presumably by adding glob-support
> to {{LocalFileSystem}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
