liutang123 commented on a change in pull request #24131: [SPARK-27192][Core]
spark.task.cpus should be less or equal than spark.executor.cores
URL: https://github.com/apache/spark/pull/24131#discussion_r269886184
##########
File path: core/src/test/scala/org/apache/spark/SparkConfSuite.scala
##########
@@ -140,13 +140,6 @@ class SparkConfSuite extends SparkFunSuite with
LocalSparkContext with ResetSyst
assert(sc.appName === "My other app")
}
- test("creating SparkContext with cpus per tasks bigger than cores per
executors") {
Review comment:
Hi, @srowen I add a UT in SparkContextSuite witch is more reasonable than
this I think.
If you think #23290 is necessary to be retained, this PR only requires
checking the local mode.
But, imagine this case:
`spark.executor.cores 1` in conf/spark-defaults.conf as default conf.
If user exec `spark-shell --master local[6] --conf spark.task.cpus=2`
command, the checking logic in #23290 will throw a exception. This logic forces
the user to set `spark.executor.cores` larger than 2 although the
`spark.executor.cores` is meaningless. So, I think we can check
`spark.task.cpus` before creating SchedulerBackend and TaskScheduler - but what
do you think？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
