dmkozh commented on pull request #14839: URL: https://github.com/apache/beam/pull/14839#issuecomment-848279088
> It is possible to use interactive runner to capture a pipeline and send as a complete beam job. Are those impacted? (/cc @KevinGG ) I'm not sure about that, does it officially support dataflow? The problem with the remote execution is that most of the global context is not transferred (such as imports, functions/classes called from transforms, global variables etc). With byref=False we 'accidentally' transfer the whole globally defined transform class, but not anything else from the global scope (so if the transform e.g. calls an import/function, it won't work remotely). So I think InteractiveRunner needs a way of transferring globals which is a) exclusive to it and b) more robust in terms of what get transferred. Maybe as a quick short-term option InteractiveRunner could set the dill byref option globally to preserve the current behavior. For all the other (non-interactive) pipelines byref=False only increases the pipeline proto size and makes local pipeline roundtrips buggy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
