Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/12494#issuecomment-212745533
@HyukjinKwon ok. make thing clearer...
@mathieulongtin wants to use `null` as option value in spark-csv from
databricks. So he did the PR to allow passing `None` from python to Scala,
instead of string "None".
The passed `None` will be `null` in Scala side.
However, passing `null` as option value in current codes will cause null
exception. Because Spark CSV data source does not handle `null`.
In this PR, I filter nulls and disallow using null as option value. It is
not @mathieulongtin wants in his original PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
