tvalentyn commented on pull request #16920:
URL: https://github.com/apache/beam/pull/16920#issuecomment-1049436020
This looks to be one of the lesser known features of Dataflow runner V1, and
is marked experimental. I wasn't aware of it. @rahuliyer95 would you be
interested in making it a non-experimental first-class citizen in Beam that
would work for all runners? I would recommend an advertisement on dev
describing your use-cases for this, settle on the expected plugin format (class
vs module to remove the ambiguity or allow both if that makes sense sometimes -
if so you could add rationale), adding unit tests and proper documentation,
removing the 'experimental' status from the doc. You can start the conversation
on dev@ with a proposal to remove the 'experimental' bit and reference your PR.
This is what the import code looks like in the Dataflow Python worker:
```
for beam_plugin in sdk_pipeline_options.get('beam_plugins', []):
try:
module_name, class_name = beam_plugin.rsplit('.', 1)
_ = getattr(importlib.import_module(module_name), class_name)
logging.info('Successfully imported beam plugin %s', beam_plugin)
except ImportError:
logging.warning('Failed to import beam plugin %s', beam_plugin,
exc_info=True)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
