Hi,
I installed airflow<python3> on my laptop and using BashOperator to login
to remote EC2 server and submit a python program. On my Ec2 machine I have
both Python2 <2.7.6> and Python3 <3.5.1>.
If I login manually to Ec2 instance and type just "python" it opens
Python3 shell.
But when I execute through BashOperator command using airflow it is going
to Python2.
Please see below command and output. The first command "python --version"
output is "{bash_operator.py:77} INFO - Python 2.7.6". for Python3
--version it gives "Python 3.4.3 " But I have 3.5.1
This is causing a bigger problem because I need to execute a Jupiter
notebook that has to process utf-8 chars. my notebook is in python3.
Python2 processing of same notebook gives utf-8 char related error and
fixing that is not possible from my side.
When I tried to execute the python program from BashOperator using
something like "python3 abc.py" it still uses python2 some where internally
so utf-8 char related error comes.
Please suggest me or point me out what I am missing. I have been trying
different options but didn't get any progress. Any help would be highly
appreciated.
Please see below log:
sparkcmd = """
ssh -i /users/msr/Downloads/EC2/abc.pem
[email protected] 'python --version;
python3 --version;'
"""
t2 = BashOperator(
task_id='ClassifyNews',
bash_command=sparkcmd,
retries=3,
dag=dag)
[2016-06-23 07:50:25,522] {models.py:1219} INFO - Executing
<Task(BashOperator): ClassifyNews> on 2016-05-04 00:00:00
[2016-06-23 07:50:25,535] {bash_operator.py:55} INFO - tmp dir root
location:
/var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T
[2016-06-23 07:50:25,536] {bash_operator.py:64} INFO - Temporary script
location
:/var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T/airflowtmpa1cd9t9s//var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T/airflowtmpa1cd9t9s/ClassifyNewsgwx9mtr5
[2016-06-23 07:50:25,536] {bash_operator.py:65} INFO - Running command:
ssh -i /users/msr/Downloads/EC2/abc.pem
[email protected] 'python --version;
python3 --version;'
[2016-06-23 07:50:25,539] {bash_operator.py:73} INFO - Output:
[2016-06-23 07:50:26,489] {bash_operator.py:77} INFO - Python 2.7.6
[2016-06-23 07:50:26,489] {bash_operator.py:77} INFO - Python 3.4.3
[2016-06-23 07:50:26,491] {bash_operator.py:80} INFO - Command exited with
return code 0
