MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource URL: https://github.com/apache/spark/pull/23325#issuecomment-450385997 Let's imagine a situation when an user uses `from_json` to parse a column with JSON records and for some of the records JSON parser wasn't able to detect any JSON tokens. It could be empty string, string with a few spaces or something else. Would it be useful to find out what particular strings the parser cannot parser? If we return such strings in the corrupt column, probably it could help the user in trouble shooting. Just returning `null`s won't give any opportunities to user to debug or put such string to a separate dataset. I think it makes sense to keep current behavior of `from_json` and revert back behavior of JSON datasource only as the PR proposes.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
