sarutak commented on a change in pull request #25487: [SPARK-28769][CORE]
Improve warning message of BarrierExecutionMode when required slots > maximum
slots
URL: https://github.com/apache/spark/pull/25487#discussion_r316738581
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -433,8 +433,10 @@ private[spark] class DAGScheduler(
* submission.
*/
private def checkBarrierStageWithNumSlots(rdd: RDD[_]): Unit = {
- if (rdd.isBarrier() && rdd.getNumPartitions > sc.maxNumConcurrentTasks) {
- throw new BarrierJobSlotsNumberCheckFailed
+ lazy val numPartitions = rdd.getNumPartitions
Review comment:
Yes they, especially `rdd.getNumPartitions`, are cheap to access. It's just
for removing unused variable and method call in case `rdd.isBarrier()` returns
`false`.
I have no strong preference for whether making them lazy, nesting if
statements or just removing `lazy` modifier.
It's just my curiosity but making those variables lazy brings another
overhead or poor readability of code?
Anyway, I'll remove `lazy`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
