Hi folks, Sometimes, it would be very useful if PTransforms had access to global pipeline options, such as various credentials, settings and so on.
Per conversation in https://issues.apache.org/jira/browse/BEAM-2572, I'd like to kick off a discussion about that. This would be beneficial for at least one major use case: support for different cloud providers (AWS, Azure, etc) and an ability to specify each provider's credentials just once in the pipeline options. It looks like the trickiest part is not to make the PTransform objects have access to pipeline options (we could possibly just modified the Pipeline.apply <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline.py#L355> method), but to actually pass these options down the road, such as to DoFn objects and FileSystem objects. I'm still in the process of reading the code and understanding of what this could look like, so any input would be really appreciated. Thank you. -- Best regards, Dmitry Demeshchuk.
