Github user yanboliang commented on the issue:
https://github.com/apache/spark/pull/16214
@mengxr I think there is no (at least I did not found) concurrency issue
when multiple executors on the same machine, since native R has lock to protect
this. You can verify it by installing package to a shared directory in a single
node in multi-thread way. However, we should recommend to install package to an
executor associated directory which means different executor has its own R lib
directory even if they are in the same machine. For YARN mode, it works well
since each executor has its own R lib directory natively.
@felixcheung I updated the examples and add corresponding session in SparkR
user guide. I'm not sure whether such an example is appropriate, since it has
many dependencies on environment. But I think for an interactive analytical
tool, installing packages across the session is not very rare. Still, I'm open
to hear your thought, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
