[
https://issues.apache.org/jira/browse/BEAM-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976838#comment-16976838
]
David Moravek commented on BEAM-8577:
-------------------------------------
This is broken in previous versions as well. Removing 2.17 tag as we still
didn't reach consensus how to properly solve this (the PR is just a workaround).
> FileSystems may have not be initialized during ResourceId deserialization
> -------------------------------------------------------------------------
>
> Key: BEAM-8577
> URL: https://issues.apache.org/jira/browse/BEAM-8577
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Affects Versions: 2.16.0
> Reporter: David Moravek
> Assignee: David Moravek
> Priority: Major
> Fix For: 2.17.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> - FileSystems use static registration using
> *FileSystems#setDefaultPipelineOptions* method.
> - *#setDefaultPipelineOptions* is called either when deserializaing
> SerializablePipelineOptions or during opening of various beam operators.
> - *FileIO#matchAll* is expanded using *Reshuffle.viaRandomKey()*.
> - Reshuffle is implemented using *.rebalance*, that doesn't have a
> "RichFunction" lifecycle, so we need to find another way to register
> FileSystems, as the deserialization may happen before other "rich operators"
> get executed on particular task manager.
> This results in random pipeline fails as the task assignment is not
> deterministic.
> We can workaround this, by registering FileSystems during coder
> deserialization.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
