sjvanrossum commented on PR #37145: URL: https://github.com/apache/beam/pull/37145#issuecomment-3675547922
I think I've reviewed a similar PR before, but I can't recall which one. This seems useful, but note that these changes only introduce a distinct cache per instance of `BigQueryWrapper`. It's worth checking where those instances are created throughout the project and whether they're shared across replicas of a DoFn or not. If every replica of a DoFn creates their own cache (e.g., deserialized from its ParDo payload, created during the setup phase of its lifecycle), then the effects of these changes are somewhat limited. If that's the case then a class method with caching behavior could improve that, but may require some changes to the cache key to avoid clashes between distinct (by value, not by reference) BigQuery clients. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
