Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/16411
I'm a little worried this is very open ended and could cause a lot of
issues with users using it wrong. This opens up customers to basically do
anything they want while launching an executor. Even not launching an executor
since really this is replacing the normal executor launch command with this
script. It relies on that customers script to actually launch the executor
based on the command passed in.
If this goes in it definitely need much better explanation and docs on how
to properly write and use it. I would rather see it being more truly of a
pre-init script then a total replacement. Perhaps the spark executor launch
command is a script that will pre-pend some users stuff but then makes sure it
still calls the normal java executor launch command.
Also what about yarn cluster mode?
Do you have test results of configuring numa that shows definite
improvements? How does this compare to Automatic NUMA balancing that I
believe is on by default in Rhel7. I realize perhaps most machines aren't
running rhel7 yet but wondering if it was tried.
does numactl require special priveleges (like root) to do certain
operations?
The script looks very basic which I understand for an example is fine but
it seems like there are definitely things missing and things people could get
wrong.
For instance, how do you handle multiple containers on a node. How does
this work when you specify an executor to have X cores.
Note I haven't done any tuning of numa myself so sorry if some of these
questions seem obvious.
How does processes with numa configured interact with processes that don't?
It seems like tuning things right could be quite hard especially if running
on something like yarn where other applications aren't using the same logic to
configure numa.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
