Thanks Bryn.
On Wed, Mar 5, 2014 at 9:00 PM, Bryn Keller <xol...@xoltar.org> wrote: > Hi Christian, > > The PYSPARK_PYTHON environment variable specifies the python executable to > use for pyspark. You can put the path to a virtualenv's python executable > and it will work fine. Remember you have to have the same installation at > the same path on each of your cluster nodes for pyspark to work. If you're > creating the spark context yourself in a python application, you can use > os.environ['PYSPARK_PYTHON'] = sys.executable before creating your spark > context. > > Hope that helps, > Bryn > > > On Wed, Mar 5, 2014 at 4:54 AM, Christian <chri...@gmail.com> wrote: > >> Hello, >> >> I usually create different python virtual environments for different >> projects to avoid version conflicts and skip the requirement to be root to >> install libs. >> >> How can I specify to pyspark to activate a virtual environment before >> executing the tasks ? >> >> Further info on virtual envs: >> http://virtualenv.readthedocs.org/en/latest/virtualenv.html >> >> Thanks in advance, >> Christian >> > >
