aleksandar-trajkovic-db commented on PR #59671: URL: https://github.com/apache/spark/pull/59671#issuecomment-6523559333
I would argue against pinning the option between schema resolution and execution. Firstly, this change complicates the code a fair bit, as the V2 path doesn't really contain a proper mechanism to propagate an option between the initial DF definition and the execution, so we need to make an exception for just this one option and just at one construction site for the JdbcOptions. Secondly, we are doing this for a pretty minor edge case, where the user flips this legacy flag between the dataframe definition and the dataframe execution. I would call this case unlikely and I wouldn't say the expected behavior necessarily is to pin the decision from the dataframe definition. Since the Spark conf affects both query execution and schema resolution, I would argue it's just as accurate that the expected behavior would be that if the user changes the flag in between, the flag would now affect query execution immediately, even on initialized dfs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
