[
https://issues.apache.org/jira/browse/BEAM-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127062#comment-17127062
]
Joar Wandborg commented on BEAM-5417:
-------------------------------------
[~kenn] [~udim] As I mentioned in
https://issues.apache.org/jira/browse/BEAM-5417?focusedCommentId=16742909&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16742909,
this bug was probably resolved in the version that
[https://github.com/apache/beam/pull/6423] was first included in, which seems
to be 2.8.0.
> FileSystems.match behaviour diff between GCS and local file system
> ------------------------------------------------------------------
>
> Key: BEAM-5417
> URL: https://issues.apache.org/jira/browse/BEAM-5417
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Affects Versions: 2.5.0, 2.6.0
> Reporter: Joar Wandborg
> Assignee: Udi Meiri
> Priority: P2
> Labels: stale-assigned
> Time Spent: 7h
> Remaining Estimate: 0h
>
> Given the directory structure:
>
> {noformat}
> .
> ├── filesystem-match-test
> │ ├── a
> │ │ └── file.txt
> │ └── b
> │ └── file.txt
> └── filesystem-match-test.py
> {noformat}
>
> Where {{filesystem-match-test.py}} contains:
> {code:python}
> from __future__ import print_function
> import os
> import posixpath
> from apache_beam.io.filesystem import MatchResult
> from apache_beam.io.filesystems import FileSystems
> BASES = [
> os.path.join(os.path.dirname(__file__), "./"),
> "gs://my-bucket/test/",
> ]
> pattern = "filesystem-match-test/*/file.txt"
> for base_path in BASES:
> full_pattern = posixpath.join(base_path, pattern)
> print("full_pattern: {}".format(full_pattern))
> match_result = FileSystems.match([full_pattern])[0] # type: MatchResult
> print("metadata list: {}".format(match_result.metadata_list))
> {code}
> Running {{python filesystem-match-test.py}} does not match any files locally,
> but does match files on GCS:
> {noformat}
> full_pattern: ./filesystem-match-test/*/file.txt
> metadata list: []
> full_pattern: gs://my-bucket/test/filesystem-match-test/*/file.txt
> metadata list:
> [FileMetadata(gs://my-bucket/test/filesystem-match-test/a/file.txt, 6),
> FileMetadata(gs://my-bucket/test/filesystem-match-test/b/file.txt, 6)]
> {noformat}
> The expected result is that a/file.txt and b/file.txt should be matched for
> both patterns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
