Github user kalpit commented on the pull request:
https://github.com/apache/spark/pull/554#issuecomment-41474348
@mateiz I ran into this when my custom RDD<String> produced nulls for some
elements within a partition/split (during compute()).
It would indeed be better to pass a null to Python and have it display it
as None. One solution is to a pick a TOKEN that we write into the tmp file and
then translate it to a "None" during read. This, however, is not failsafe
because there is a remote possibility of string data being identical to the
TOKEN. Perhaps we could address that by fencing regular data by a special
character and treating data lacking that fence as tokens.
In any case, the above solution (or an alternative) would be a relatively
larger change, and I preferred fixing at least the NPEs in PythonRDD for short
term (stack trace is in JIRA ticket).
What do you think ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
