Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/7001#issuecomment-115764745
So I took a look at this and I managed to figure out why the tests don't
work but why it works from sparkR shell. Its due to the fact we have two
different ways in which things are initialized
1. For the sparkR shell (and RStudio), the R process comes up first and
then the JVM is launched when the spark context is created. In this case if we
pass in `sparkJars` they are added to the spark-submit command and things work
fine.
2. For the unit tests (and any script run with spark-submit) the JVM comes
up first and forks the R process. In this case adding sparkJars to the context
has no effect as the JVM is already running. In this case the user needs to run
`sparkR --jars <jarFile>` to get things to work correctly.
For the unit tests we can't test the first case as it is a batch script.
But we can test the second case by actually running the sparkR script from the
unit test. This would be something similar to
https://github.com/apache/spark/blob/master/python/pyspark/tests.py#L1641
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
