skambha commented on issue #23206: [SPARK-26249][SQL] Add ability to inject a rule in order and to add a batch via the Spark Extension Points API URL: https://github.com/apache/spark/pull/23206#issuecomment-452486057 Thanks Xiao for the feedback. I am evaluating the options. Xiao > "The order matters in extreme cases. Normally, for those very advanced users, I found they Xiao > even extend the SparkOptimizer." From a quick look, it looks like this will need the user to extend BaseSessionStateBuilder and then also extend the SparkSession so that it can return the SessionState that has the user specified(extended) optimizer. In this approach then, it sounds like existing apps that use the SparkSession.Builder to create a SparkSession will require changes.. so it is not as user-friendly like the extension points framework that allows the apps to remain as is and inject using the Builder’s withExtensions method. Does that sound similar to the usecases you mention where users extend the SparkOptimizer? If not, can you please elaborate on the recommended end user(client) usage. Thanks
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
