Topic: do you need to add additional configuration for Python version switching and path configuration switching
CCweixiao_ intsig_ leojie: @ peacewong@WDS Here, the user selects the python version and the corresponding Python path After restarting the engine, I found that I had been using the path of python2 to execute the code, The version switching and path configuration of python2 and python3 are still not very transparent to me. I don't know what you did peacewong@WDS : The python switch of spark is mainly based on the spark in your diagram python. Version, you can specify a path, or you can directly use Python and python 3. Then, when starting the Python interpreter, go back to set env to reset pyspark_ Python environment variable So you only need to set spark python. Just version CCweixiao_ intsig_ leojie: spark. python. Version I set the acanda path for python3python3 through spark pyspark. Python biography? The path of python2 is also the virtual environment path of acanda peacewong@WDS : All through spark python. Just pass the version to the absolute path CCweixiao_ intsig_ leojie: It feels like it can be optimized for the purpose of planning the version. Therefore, the code also adds judgment. If it is Python 3, the variable pyspark will be read_ PYTHON3_ Path to get the specific path. Initial configuration, because there are Python and python 3 commands on your machine by default, which is really confusing Add pyspark_ DRIVER_ Python3, or add spark python. Path, get pyspark if you don't pass it_ DRIVER_ Python path peacewong@WDS : Based on the current spark python. Version may be better because compatibility upgrade should be considered. This parameter is available on the management console by default. The main modification point may be to consider setting pyspark when starting the Python interpreter_ Set Python so that the driver and executor will be consistent. Summary: do not add additional configuration of Python version related paths for the time being
