Github user xiaochang-wu commented on the issue:
https://github.com/apache/spark/pull/16411
@tgravescs Thanks for your comments. There are two things we have tried.
1) To add a prefix command on executor launch
I agree this opens a door for user to do anything for launching the
executor. This patch is intended for profiling and debugging. May not fit for
production. I am not sure it's the best form to implement, it fixed our problem
quickly.
2) NUMA
The script attached about NUMA is only an example to show how to use this
patch. User can customize it to fulfill their specific needs. Automatic NUMA
balancing is by default enabled on our system. As mentioned in the original
Redhat slides, It can only deal with certain cases and still can not beat
manual pinning. From our experiments, not all cases have big NUMA penalties. We
should use some platform tools such as Intel VTune to identify if there is a
NUMA problem and tune case by case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
