DaveDeCaprio opened a new pull request #23865: [SPARK-26957][SCHEDULER] Config properties for spark dynamic scheduler pools URL: https://github.com/apache/spark/pull/23865 ## What changes were proposed in this pull request? This fix addresses the issue raised in SPARK-26957. If a spark.scheduler.pool is set to a name that was not a preconfigured pool, the pool was always created with the default properties. This fix allows the default scheduler pool properties to be configured through configuration parameters. Previously they were hardcoded. The fix is fully backwards compatible because the configuration properties default to the existing hardcoded values. This PR just allows those values to be updated. The specific use case where we needed this was explained in SPARK-26957 ## How was this patch tested? A unit test was added to PoolSuite that sets the config values, generates a pool, and checks that the pool is using the configured values. An existing test verifies that with no special configuration settings, the original defaults are used. In addition to this unit testing, we are currently using a version of this patch in production on AWS EMR for Spark 2.4.0. This contribution is my original work and that I license the work to the project under the project’s open source license.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
