cloud-fan commented on pull request #30106: URL: https://github.com/apache/spark/pull/30106#issuecomment-721823852
The schema inference is already in the DS v2 API `TableProvider` which is not bound to file source. I think this feature "skip schema inference on write" should not bound to file source as well. Instead of adding a marker trait, I think we can just reuse `TableProvider.supportsExternalMetadata`, which indicates that the schema inference is expensive. If `supportsExternalMetadata == true`, the `DataFrameWriter.save` should not infer schema and just pass the input schema to `TableProvider`. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
