I have checked that cloudpickle (an alternative to dill) is able to pickle and unpickle typing types on Python 3.5, 3.6, which seems to be a recent change, see: https://github.com/cloudpipe/cloudpickle/issues/63#issuecomment-501624383.
I am evaluating cloudpickle as a potential avenue to address several other issues we found in Beam while working on Python 3 support, such as: https://issues.apache.org/jira/browse/BEAM-6522 https://issues.apache.org/jira/browse/BEAM-7284 https://issues.apache.org/jira/browse/BEAM-5878?focusedCommentId=16834554 https://github.com/uqfoundation/dill/issues/300 https://issues.apache.org/jira/browse/BEAM-7540 Once I have more information on cloudpickle vs dill in Beam, I'll bring it to the mailing list. On Wed, May 15, 2019 at 5:25 AM Robert Bradshaw <[email protected]> wrote: > (2) seems reasonable. > > On Tue, May 14, 2019 at 3:15 AM Udi Meiri <[email protected]> wrote: > > > > It seems like pickling of typing types is broken in 3.5 and 3.6, fixed > in 3.7: > > https://github.com/python/typing/issues/511 > > > > Here are my attempts: > > https://gist.github.com/udim/ec213305ca865390c391001e8778e91d > > > > > > My ideas: > > 1. I know that we override type object handling in pickler.py > (_nested_type_wrapper), and perhaps this mechanism can be used to pickle > typing classes correctly. The question is how. > > > > 2. Exclude/stub out these classes when pickling a pipeline - they are > only used for verification during pipeline construction anyway. This could > be a temporary solution for versions 3.5 and 3.6. > > > > Any ideas / opinions? >
