claudevdm commented on PR #36359: URL: https://github.com/apache/beam/pull/36359#issuecomment-3362359422
Historically dill was used to pickle the "special types" https://github.com/apache/beam/pull/14180, and values are pickled separately to reconstruct the instance Not sure if you remember that PR, it has been a while, but it seems you were satisfied with the determinism of pickling special types like namedtuples, dataclasses etc. > Ah, so you're saying that the pickled bytes are used for coder comparison for update compatibility? That's potentially dangerous (pickles have low determinism guarantees) but I suppose works most of the time. In that case, yes, we'd likely need to guard this with a flag. Thats right, whenever we change how the types are pickled, it breaks update compatibility for Shuffles in streaming pipelines, so we have to guard it by a flag. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
