Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13824
I'm fine with the change to allow this overriding the existing
PYSPARK_PYTHON and others, but on the change from addPathToEnvironment I'm
against. The problem is that some env variables don't take this but others
do. I think it was this way just incase the variable required a classpath type
combining, for instance LD_LIBRARY_PATH, or I think any other env variables
that have "PATH" in the name of them. We don't have a current way to
differentiate when they should be combined with existing vs just overwritten.
If you want to change that I would say we need a way to differentiate. So
whether that is looking for "PATH" in the env variable name and appending or
having a config or something that says for these variables use append else
don't. Probably should be a separate jira for this one though.
Note here is jira very similar in MapReduce:
https://issues.apache.org/jira/browse/MAPREDUCE-6491
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
