Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/3816#issuecomment-68285949
Well, what I'm saying is to look at how it affects the rest of the
scheduler. That was set to PROCESS_LOCAL there for a reason, it wasn't a typo.
It was to make sure that launching a no-pref task doesn't then cause you to
increase your allowed locality level and miss waiting for other local ones. I'd
also like to see what performance different this makes in the original case,
and why it was a problem there (e.g. was this an InputFormat with no locality
info at all or something).
One fix by the way may be to not count NO_PREF launches at all when
deciding how to update delay scheduling variables, but even then it's good to
understand what this was doing and make sure it won't break it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
