Github user NiharS commented on the issue:
https://github.com/apache/spark/pull/22192
I'll change the config documentation to specify YARN only, hopefully it's
not a huge issue.
It seems like the line
`Thread.currentThread().setContextClassLoader(replClassLoader)` is causing the
pyspark failures, they pass when I remove it. I'm looking at the test cases but
I really don't see how this is affecting them...it seems that in both test
cases, the DStreams monitor a directory but don't pick up the changes they're
supposed to, and just time out. I checked that I can bypass this issue by
changing back to having the plugins loaded on a separate thread (and setting
that thread's contextClassLoader instead of the current thread) and it passes
tests and continues to work. That said this issue does seem to be indicative of
some problem in pyspark streaming
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
