claudevdm commented on PR #36359: URL: https://github.com/apache/beam/pull/36359#issuecomment-3362095872
Hey @robertwb , part of the effort to relax the dependency on dill is using cloudpickle to pickle deterministic types (instead of using dill) https://github.com/apache/beam/blob/d78d004d0a5846547381c9476d6732846365497c/sdks/python/apache_beam/coders/coder_impl.py#L563 I did not disable the compression/base64 encoding in this case because a deterministic type only has to be pickled/unpickled once per process (the pickled type is deterministic). https://github.com/apache/beam/pull/36032 This is to say I think removing base64 encoding will break update compatibility, because pickled types will not be base64 encoded anymore. Can we guard this behind an update compat flag? I can make this change. Also, do you think its worth skipping the compression step in the same change for pickling these "special" types? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
