Hey Users, I want to run spark job from virtual environment using Python.
Please note I am creating virtual env (using python3 -m venv env) I see that there are 3 variables for PYTHON which we have to set: PYTHONPATH PYSPARK_DRIVER_PYTHON PYSPARK_PYTHON I have 2 doubts: 1. If i want to use Virtual env, do I need to point python path of virtual environment to all these variables? 2. Should I set these variables in spark-env.sh or should I set them using export statements. Regards Rajat
