[
https://issues.apache.org/jira/browse/BEAM-7998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914688#comment-16914688
]
Jerome MASSOT commented on BEAM-7998:
-------------------------------------
Hi Pablo,
complement of information : the MatchFiles() and ReadMatches() send me back
several times the same file ... with different metadata.path attached...
I know it because I hash the metadata.path returned to used it as PrimaryKey in
a SQL database... and I cannot remove the duplicates because the PrimaryKey of
duplicates are different...
Using hash() implementation in Python, the only possible for different hashed..
is a different metadata.path...
Good luck
Best regards
Jerome
> MatchesFiles or MatchAll seems to return seveval time the same element
> ----------------------------------------------------------------------
>
> Key: BEAM-7998
> URL: https://issues.apache.org/jira/browse/BEAM-7998
> Project: Beam
> Issue Type: Bug
> Components: io-py-files
> Affects Versions: 2.14.0
> Environment: GCP for storage, DirectRunner and DataflowRunner both
> have the problem. PyCharm on Win10 for IDE and dev environment.
> Reporter: Jerome MASSOT
> Assignee: Pablo Estrada
> Priority: Major
>
> Hi team,
> when I use MatcheFiles using wildcard and files located in a GCP bucket, the
> MatcheFiles transform returns several times (at least 2) the same file.
> I have tried to follow the stack, and I can see that the MatchesAll is called
> twice when I run the pipeline on a debug project where a single element is
> present in the bucket.
> But I am not good enough to say more than that. Sorry.
> Best regards
> Jerome
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
