cloud-fan commented on PR #40699: URL: https://github.com/apache/spark/pull/40699#issuecomment-1517277895
@sadikovi CSV does not accept struct/array/map columns, your example generates a string column and writes it to CSV. That said, any behavior change of that "string generation" function will change the value we write to CSV, but I wouldn't call it a CSV behavior change. It's intentional that the cast struct/array/map to string behavior is changed w.r.t. nulls. If third-party libraries rely on this behavior, then we need to revisit this PR. BTW, I think PR description should not say it's for consistency with spark-sql shell. `df.show` displays top-level null value as `NULL`, but inner field null value as `null`. This is the major motivation of the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
