Github user chaoslawful commented on the issue:
https://github.com/apache/spark/pull/18277
Well, the difference comes from repr()'s divergent default behaviors
between Python2 and Python3. And the previous code does no better than the
patched one but causing troubles while processing unicode strings.
On the other hand, pipe() action involved implicit serialization from any
type to bytes by its definition, so IMHO the application itself should take
care of consistent serialization/deserialization of data before/after pipe()
action, IF it wants to always get the same behavior in different environments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
