Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20648
I was just double checking the current status for both CSV and JSON:
Seems CSV fills up the partial results with an exception (which is caught
by permissive mode with the corrupt record text) when the length of schema is
mismatched whereas JSON just allows this case without the exception (and also
without the corrupt record text).
and ... both CSV and JSON don't fill the partial results when the type of
the field is mismatched but this change supports this case in JSON.
cc @cloud-fan too. I remember we had a short talk about this partial
results before. Do you think we should produce partial results in both CSV and
JSON for mismatched types?
Looks we are currently not doing this for both CSV and JSON when the types
are mismatched, if I haven't missed something.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
