panbingkun commented on PR #44665: URL: https://github.com/apache/spark/pull/44665#issuecomment-1925639982
> 1. Does CSV source support this? I think it's disallowed > 2. If we want to support this it should better be able to read it back. Problem is that CSV spec doesn't have this. > > Should probably just disallow this case I have privately discussed this issue with @LuciferYang , and in response to this situation, we actually have two options: - In `to_csv` When encountering `complex types` in CSV, such as `ArrayType`, `MapType`, `StructType`, we can prompt an error indicating that it is not supported. **However, in this case, in the `df. show()`, it can actually display complex types, which looks weird.** - Another way is for us to support it. Indeed, CSV spec doesn't have this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
