Github user superbobry commented on the issue:
https://github.com/apache/spark/pull/21180
The only workaround I know is to manually remove the __reduce__ method for
each namedtuple class in use. This has to be done on both the driver and the
executors.
> The fix itself looks not the clean as well anyway for now.
Could you kindly elaborate on the parts you'd like me to improve? A proper
fix, I think, is to just remove the patch altogether, but this looks like a
bigger change than making it work for more use-cases.
> Since 2.4.0 is being closed, we will probably avoid risky changes.
Will 2.4.0 be the last release in 2.X branch? If yes, then targeting 3.0.0
sounds reasonable, otherwise, I'd very much prefer to have it in the release
following 2.4.0.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
