JozoVilcek commented on issue #33965: URL: https://github.com/apache/beam/issues/33965#issuecomment-2667839176
@liferoad I wonder why is this `closed` as it is still being discussed. Perhaps unwanted closed? @Abacn , I admit my use-case is not a standard one. I am in a mixed pure Spark and Beam on Spark runner environment. I did came across this when in pure spark I am trying to reuse filesystem utility and was calling `FileSystems.setDefaultPipelineOptions()` manually. I was trying to use `registerFileSystemsOnce()` but that will not help me since `setDefaultPipelineOptions()` is called from inside `SerializablePipelineOptions`. I understand that API is internal and subject to change. But still I believe there is race condition in the init. @Abacn , can you share how does workers ensure this is consistently initialised under multithreaded execution. Maybe it can inspire me with some thoughts. I was not able to find it hence I believe that in Beam the race is there perhaps too just initi sequence is more loaded therefore from initial call to `setDefaultPipelineOptions()` to actual user code using the filesystem takes more time and registration finished. My code path is much much shorter and surface this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
