scwhittle commented on PR #31356: URL: https://github.com/apache/beam/pull/31356#issuecomment-2175988311
It would be pretty easy to do so but it could still cause latency regressions on pipelines that have a global side input that is not refreshing. I was looking into doing so only if the pcollection generating the side input was unbounded as a non-updating side input seems like it would be bounded. I was using the AsSideInput [pcollection](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pvalue.py#L363) but then it looked like some additional plumbing was necessary to keep this metadata in [SideInputData](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pvalue.py#L451) which is what is unpickled and used for execution. I didn't get a chance to get it all working yet. @robertwb Do you forsee issues with that approach before I work further on completing it? IIUC the pickling just needs to be consistent between pipeline submission and execution so it wouldn't be an update compatibility issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
