Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14124
@cloud-fan If nullability should be not ignored, then I can fix this PR to
make them consistent to not ignoring it (and of course I will try to identify
the related problems). In this case, I will work on what @gatorsmile pointed
out in https://github.com/apache/spark/pull/14124#issuecomment-231594416 about
JSON (and will check the other data sources as well).
I will follow your decision.
To cut the all comments above short, (for other reviewers),
- The purpose of this PR is whether it should force all schema to nullable
schema or not.
- This is already happening with normal reading and writing for data
sources based on `FileFormat`.
- This is for both inferred/read schema and user-given schema.
- For `json(rdd: RDD[String])` API, this is not hapenning.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
