cloud-fan commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource URL: https://github.com/apache/spark/pull/23325#issuecomment-448463474 > in the PERMISSIVE mode return a Row with nulls for all fields if specified root type is StructType (bad record string is placed to the corrupt column if it is specified), and null for ArrayType and MapType. I think this is the most arguable part. The current behavior looks not reasonable, I can think of 2 options: 1. always return null if the token is empty, no matter it's row or array or map. 2 never return null if the token is empty. For struct type, return a row with all null fields. For array/map, return empty array/map. @HyukjinKwon @MaxGekk any preference?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
