stczwd commented on a change in pull request #28048:
[SPARK-31142][PYSPARK]Remove useless conf set in pyspark context
URL: https://github.com/apache/spark/pull/28048#discussion_r399965407
##########
File path: python/pyspark/context.py
##########
@@ -181,11 +181,6 @@ def _do_init(self, master, appName, sparkHome, pyFiles,
environment, batchSize,
self.appName = self._conf.get("spark.app.name")
self.sparkHome = self._conf.get("spark.home", None)
- for (k, v) in self._conf.getAll():
Review comment:
> Are you saying you're relying on an environment that was not set by
`spark.executorEnv.*`?
No. We are just trying to be compatible with the user's environment variable
parameters in spark.executorEnv. *.
As for example in the above:
```bash
# these two has same results
spark.executorEnv.LD_LIBRARY_PATH=$PYTHONHOME/lib:$LD_LIBRARY_PATH;
spark.executorEnv.LD_LIBRARY_PATH =./python/lib:/usr/native_lib;
# this is also supported if user doesn't need $LD_LIBRARY_PATH
spark.executorEnv.LD_LIBRARY_PATH =./python/lib
```
We do not prevent users from overriding environment variables, we just try
to meet users' needs for using cluster environment variables or their
environment variables, like linux.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
