superbobry commented on issue #23008: [SPARK-22674][PYTHON] Removed the namedtuple pickling patch URL: https://github.com/apache/spark/pull/23008#issuecomment-451445498 Oh, sorry, I missed that you propose to keep the hack but make it opt-in. I suspect that serializability of REPL-defined namedtuples affects only a small fraction of users. Therefore, removing the hack is an acceptable behaviour change (cc @holdenk) . We could clearly document this in the ->3.X migration document and potentially enable "cloudpickle" by default when PySpark is running in an interactive mode. Keeping the hack and adding a flag on top does not fix the problematic behavior and does not make the failures any easier to diagnose.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
