Lordworms commented on issue #11150: URL: https://github.com/apache/datafusion/issues/11150#issuecomment-2241200238
> The format factories can optionally carry modified default write options. This is only used for DataFrame::write_* methods, so that they do not need to pass strings to modify write behavior. > > https://github.com/apache/datafusion/blob/827d0e3a29c0ea34bafbf03f5102407bd8e9b826/datafusion/core/src/datasource/file_format/csv.rs#L60-L62 > > The existing SerDe implementation does not handle the case when the format factory carries options itself. This will prevent systems like Ballista from correctly passing around options in its DataFrame::write_* methods. Here are the lines that are not fully implemented: > > https://github.com/apache/datafusion/blob/827d0e3a29c0ea34bafbf03f5102407bd8e9b826/datafusion/proto/src/logical_plan/file_formats.rs#L74-L89 > > You can see that the encoding method does not actually write any bytes, and the decoding method just initializes a default struct, ignoring any written bytes. let me take a look and > The format factories can optionally carry modified default write options. This is only used for DataFrame::write_* methods, so that they do not need to pass strings to modify write behavior. > > https://github.com/apache/datafusion/blob/827d0e3a29c0ea34bafbf03f5102407bd8e9b826/datafusion/core/src/datasource/file_format/csv.rs#L60-L62 > > The existing SerDe implementation does not handle the case when the format factory carries options itself. This will prevent systems like Ballista from correctly passing around options in its DataFrame::write_* methods. Here are the lines that are not fully implemented: > > https://github.com/apache/datafusion/blob/827d0e3a29c0ea34bafbf03f5102407bd8e9b826/datafusion/proto/src/logical_plan/file_formats.rs#L74-L89 > > You can see that the encoding method does not actually write any bytes, and the decoding method just initializes a default struct, ignoring any written bytes. sure, I'll take a look then -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
