[GitHub] spark issue #22849: [SPARK-25852][Core] we should filter the workOffers with...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/22849 It may happen that a busy executor is marked as lost and later it re-register to the driver, in that case currently we call `makeOffers()` and that will add the executor into `TaskSchedulerImpl.hostToExecutors`. This is bad implementation here since it shall not have depend on the `makeOffers()` function to update a unrelated protected val, but that's what we have for now. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22849: [SPARK-25852][Core] we should filter the workOffers with...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/22849 What do you mean by "better performance" ? If that means we can spend less time on `TaskSchedulerImpl.resourceOffers()` then I agree it's true, but AFAIK it's never reported this can be a bottleneck of the whole cluster, so maybe the perf gain is trivial here. If you expect better task distribution over existing executors then I don't see any case can be improved by this proposed change. Please correct me if I'm wrong. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22849: [SPARK-25852][Core] we should filter the workOffers with...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22849 Yes, that is the comment I have been referring to. So it seems you can't filter, right? it's not scheduling work here. @jiangxb1987 do you know this part of the code? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22849: [SPARK-25852][Core] we should filter the workOffers of w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22849 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22849: [SPARK-25852][Core] we should filter the workOffers of w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22849 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22849: [SPARK-25852][Core] we should filter the workOffers of w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22849 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22849: [SPARK-25852][Core] we should filter the workOffers of w...
Github user zuotingbing commented on the issue: https://github.com/apache/spark/pull/22849  --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
