Github user shivaram commented on the issue:
https://github.com/apache/spark/pull/16312
I looked at this more closely and I think I found the problem - Not sure
its easy to fix though.
What I traced here is:
- When we call sparkR.session.stop and sparkR.session the same JVM backend
is reused and only the SparkContext is stopped / recreated
- Now the problem happens when we call `read.ml` to read a model after
creating a new SparkSession. This in turn calls into RWrappers[1] which has an
`sc` member variable
- My understanding is that the `sc` member variable is bound the first time
we create a SparkSession and when we stop, restart it has a handle to the stale
SparkContext
- Thus we see errors where it says `Cannot call methods on a stopped
SparkContext`
I think the right fix here is to pass along a SparkContext into RWrappers
and not rely on a prior initialization. However I'm not sure why that design
decision was made before, so maybe I'm missing something.
[1]
https://github.com/apache/spark/blob/f252cb5d161e064d39cc1ed1d9299307a0636174/mllib/src/main/scala/org/apache/spark/ml/r/RWrappers.scala#L36
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
