scwhittle commented on issue #35934: URL: https://github.com/apache/beam/issues/35934#issuecomment-5677555520
I believe there is not real guarantee about side-input refreshes in the beam model it is just best-effort in the dataflow runner by sprinkling cache timeouts around to handle side-inputs refreshing. A side-input observed for a window is always "valid" for that window from the model. @kennknowles to keep me honest. I originally just removed this dofninvoker cache but that led to 25% regression https://github.com/apache/beam/pull/31356#issuecomment-2125627404 what about we: 1. add this behavior to clear at the end of a bundle behind an option 2. auto-enable the option if the side input cache is enabled Then those who want refreshing side inputs can either enable the option or enable the cache. And if we roll the cache out completely everyone will pick up the improvement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
