Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/13824
I think there are two ways to solve the problem that might be a little
better...
The first is to try to keep the current behavior. If, in L734 (where you're
removing the current code that checks the `appMasterEnv` conf), you make a copy
of the keys that were added to the `env` variable, you can then apply
`addPathToEnvironment` just to the keys that are there. That means that the
code will merge any user configuration with env variables created by Spark
itself; otherwise it will use the user's override.
The second is to read certain env variables using a special method that
first looks at `spark.yarn.appMasterEnv.FOO` and if it doesn't exist,
`sys.env("FOO")`. Then you could modify the code that currently reads
`PYSPARK_DRIVER_PYTHON` and friends using that new method, instead of directly
peeking at `sys.env`. You could also apply that new method to the code that
currently reads `PYTHONPATH`.
I think the latter is a better solution than you currently have, since it
avoids hardcoding these env variable names in more places.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
