squito commented on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location URL: https://github.com/apache/spark/pull/26633#issuecomment-562346943 are you sure that in your case the slowdown is even caused by SPARK-18886? Even when that is solved, you could still get end up with *one* 3s wait for almost all tasks. I really do see how this can help, I see why folks want this. But my hesitance is that we're going to start putting in these random changes to delay scheduling, which will make the code even harder to understand; users will end up with even more knobs to tune; and we may be stuck with this even after SPARK-18886 because it would still be a performance regression against this change. I agree with Tom's point -- I don't see how we know that ignoring locality waits is right for just this *one* RDD but not for others. Though I want the default locality wait set to 0, I could see a cluster admin wanting to increase the locality wait because they know their cluster is very network constrained. In fact this may be against the wishes of of one particular spark application, but still best for the cluster as a whole. In that case, you really might want a 3s wait on LocalShuffledRowRDD
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
