maryannxue commented on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location URL: https://github.com/apache/spark/pull/26633#issuecomment-558379888 I don't think there is a need to restrict it. Every RDD should "know" their own locality preference as well as the penalty for a locality miss. If we ever needed to make sure the WILDCARD is being used properly, we would have to worry about whether other regular preferred locations are returned correctly and truly reflect their best possible locality choice. That said, the 3 minute global locality wait time is not gonna work for all cases, and yet the WILDCARD location alone is not fine-grained enough either. Ideally we should have penalty, or say, the importance of locality, "encoded" with location itself, in the form of wait time, e.g., "wildcard, 2s", so that the task can still try to wait a minimum time before getting randomly assigned. However, this would involve big changes to the current Spark scheduler.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
