Hi, Please ignore this.
This is happening because remote EC2 has problem with some symbolic links pointing python. I have to give the full path of python in Bash command. Thanks, Madhu On Thu, Jun 23, 2016 at 8:15 AM, Msr Msr <[email protected]> wrote: > Hi, > > I installed airflow<python3> on my laptop and using BashOperator to login > to remote EC2 server and submit a python program. On my Ec2 machine I have > both Python2 <2.7.6> and Python3 <3.5.1>. > > If I login manually to Ec2 instance and type just "python" it opens > Python3 shell. > > But when I execute through BashOperator command using airflow it is going > to Python2. > > Please see below command and output. The first command "python --version" > output is "{bash_operator.py:77} INFO - Python 2.7.6". for Python3 > --version it gives "Python 3.4.3 " But I have 3.5.1 > > This is causing a bigger problem because I need to execute a Jupiter > notebook that has to process utf-8 chars. my notebook is in python3. > Python2 processing of same notebook gives utf-8 char related error and > fixing that is not possible from my side. > > When I tried to execute the python program from BashOperator using > something like "python3 abc.py" it still uses python2 some where internally > so utf-8 char related error comes. > > Please suggest me or point me out what I am missing. I have been trying > different options but didn't get any progress. Any help would be highly > appreciated. > > Please see below log: > > sparkcmd = """ > ssh -i /users/msr/Downloads/EC2/abc.pem > [email protected] 'python > --version; python3 --version;' > """ > > t2 = BashOperator( > task_id='ClassifyNews', > bash_command=sparkcmd, > retries=3, > dag=dag) > > [2016-06-23 07:50:25,522] {models.py:1219} INFO - Executing > <Task(BashOperator): ClassifyNews> on 2016-05-04 00:00:00 > [2016-06-23 07:50:25,535] {bash_operator.py:55} INFO - tmp dir root > location: > /var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T > [2016-06-23 07:50:25,536] {bash_operator.py:64} INFO - Temporary script > location > :/var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T/airflowtmpa1cd9t9s//var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T/airflowtmpa1cd9t9s/ClassifyNewsgwx9mtr5 > [2016-06-23 07:50:25,536] {bash_operator.py:65} INFO - Running command: > ssh -i /users/msr/Downloads/EC2/abc.pem > [email protected] 'python --version; > > python3 --version;' > > [2016-06-23 07:50:25,539] {bash_operator.py:73} INFO - Output: > [2016-06-23 07:50:26,489] {bash_operator.py:77} INFO - Python 2.7.6 > [2016-06-23 07:50:26,489] {bash_operator.py:77} INFO - Python 3.4.3 > [2016-06-23 07:50:26,491] {bash_operator.py:80} INFO - Command exited with > return code 0 >
