pan3793 commented on PR #48822: URL: https://github.com/apache/spark/pull/48822#issuecomment-2476404339
Wait, I forget the [`SessionConfigSupport`](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SessionConfigSupport.java) In fact, I submitted a [PR](https://github.com/apache/iceberg/pull/7732) to Iceberg a patch to support this feature, but unfortunately, this patch doesn't seem to be getting attention, @szehon-ho do you think we can re-open this PR and get it in? If so, the assumption would not hold. > ... that only one can be set and we should define the priority, I think it should be 1. `options` from SQL `options` 2. `options` from DataFrame API 3. `options` from session configuration Currently, if there are duplicated options, 2 overrides 3, see https://github.com/apache/spark/blob/c1968a1bd85fa6f1a274849427105fa5327ca7e0/sql/core/src/main/scala/org/apache/spark/sql/internal/DataFrameWriterImpl.scala#L141-L142 @cloud-fan, do you think the proposed priority makes sense? or any new ideas? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
