Github user Tagar commented on the issue:
https://github.com/apache/spark/pull/23113
`auth_token` is only used in accumulators update server
https://github.com/apache/spark/search?q=auth_token&unscoped_q=auth_token
- relatively new code that was added in this commit -
https://github.com/apache/spark/commit/15fc2372269159ea2556b028d4eb8860c4108650#diff-c3339bbf2b850b79445b41e9eecf57c4R249
We use Zeppelin 0.8, and it was working fine before 2.3.2 upgrade - all
other releases of Spark 2.2, 2.1 and 2.3.1 were working fine.
If pySpark context is created using existing jvm gateway, like Zeppelin
does, then making `auth_token` mandatory seems to be a breaking change? A
temporary workaround for 2.3 and 2.4 branches might be just adding an `if`
condition here
https://github.com/apache/spark/blob/master/python/pyspark/accumulators.py#L254
and bypass `auth_token` check if spark context doesn't have it set?
I agree it's a good idea to add a check if `auth_token` exists in existing
jvm gateway, and maybe silently set a new `auth_token` or fail spark context
creation like you were saying, but again it's a breaking change for a next
major release?
Also I still don't know why rerunning same code fixes the issue. It might
be just due to the fact that there are no updates are coming to the
accumulators update server - it only happens on 1st run? Then the problem just
lurking until next update will come to the accumulators update server.
Thoughts? Thank you both.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
