Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19840
I think in YARN we have several different ways to set `PYSPARK_PYTHON`, I
guess your issue is that which one should take priority?
Can you please:
1. Define a consistent ordering for such envs, which one should take
priority (spark.yarn.appMasterEnv.XXX or XXX), and document it.
2. Check if it works as expected for
`spark.yarn.appMasterEnv.PYSPARK_PYTHON` and `spark.executorEnv.PYSPARK_PYTHON`.
I guess for other envs, it will also suffer from this problem, am I right?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
