sunchao commented on code in PR #46782: URL: https://github.com/apache/spark/pull/46782#discussion_r1618262152
########## docs/core-migration-guide.md: ########## @@ -48,7 +48,9 @@ license: | - Since Spark 4.0, the MDC (Mapped Diagnostic Context) key for Spark task names in Spark logs has been changed from `mdc.taskName` to `task_name`. To use the key `mdc.taskName`, you can set `spark.log.legacyTaskNameMdc.enabled` to `true`. -- Since Spark 4.0, Spark performs speculative executions less agressively with `spark.speculation.multiplier=3` and `spark.speculation.quantile=0.9`. To restore the legacy behavior, you can set `spark.speculation.multiplier=1.5` and `spark.speculation.quantile=0.75`. +- Since Spark 4.0, Spark performs speculative executions less aggressively with `spark.speculation.multiplier=3` and `spark.speculation.quantile=0.9`. To restore the legacy behavior, you can set `spark.speculation.multiplier=1.5` and `spark.speculation.quantile=0.75`. + +- Since Spark 4.0, when configurations are provided to `spark-submit` or `spark-shell` via `--properties-file` parameter, Spark will also load extra configurations from `SPARK_CONF_DIR/spark-defaults.conf`. Those configurations defined in the former take higher precedence over those from the latter. Review Comment: Actually the PR only changes `SparkSubmitArguments` so others in the above are not affected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
