[
https://issues.apache.org/jira/browse/AIRFLOW-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Kaushik updated AIRFLOW-6124:
----------------------------------
Attachment: Error.png
Docs Text:
Description:
My dag has three SparkSubmitOperators and throws this error
{{```[2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,547] \{base_hook.py:84} INFO - Using connection to: id: spark_default.
Host: yarn, Port: None, Schema: None, Login: None, Password: None, extra:
\{'deploy-mode': 'cluster', 'spark-binary': '/usr/bin/spark2-submit/'}
[2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,548] \{spark_submit_hook.py:302} INFO - Spark-Submit cmd:
['spark-submit', '--master', 'yarn', '--conf', 'spark.yarn.maxAppAttempts=2',
'--files', '/home/centos/airflow/dags/conf/aws_keys.pickle', '--num-executors',
'4', '--executor-cores', '4', '--executor-memory', '20G', '--name',
'universal_cross_matrix_joins_AUS_27-11-2019', '--deploy-mode', 'cluster',
'/home/centos/airflow/scripts/near-airflow-repo/scripts/universal_cross_matrix/join_ifa_ncid.py',
'--input_date', '27-11-2019', '--country', 'AUS', '--tenant_id', '63ce7457']
[2019-11-29 12:58:54,553] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,553] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving
current directory: getcwd: cannot access parent directories: No such file or
directory
[2019-11-29 12:58:54,555] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,555] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving
current directory: getcwd: cannot access parent directories: No such file or
directory
[2019-11-29 12:58:54,557] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,557] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving
current directory: getcwd: cannot access parent directories: No such file or
directory
[2019-11-29 12:58:54,560] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,560] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
current directory: getcwd: cannot access parent directories: No such file or
directory
[2019-11-29 12:58:54,563] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,563] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
current directory: getcwd: cannot access parent directories: No such file or
directory
[2019-11-29 12:58:54,565] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,564] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
current directory: getcwd: cannot access parent directories: No such file or
directory
[2019-11-29 12:58:54,567] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,567] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving
current directory: getcwd: cannot access parent directories: No such file or
directory
[2019-11-29 12:58:54,569] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,569] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
current directory: getcwd: cannot access parent directories: No such file or
directory
[2019-11-29 12:58:54,571] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,571] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
current directory: getcwd: cannot access parent directories: No such file or
directory
[2019-11-29 12:58:54,574] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,574] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
current directory: getcwd: cannot access parent directories: No such file or
directory
[2019-11-29 12:58:54,576] \{logging_mixin.py:112} INFO - [2019-11-29
12:58:54,576] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
current directory: getcwd: cannot access parent directories: No such file or
directory```}}
was:My dag has three SparkSubmitOperators and throws this error
> tmp folder gets deleted, shell-init: error retrieving current directory
> -----------------------------------------------------------------------
>
> Key: AIRFLOW-6124
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6124
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
> Affects Versions: 1.10.6
> Reporter: Ravi Kaushik
> Priority: Critical
> Attachments: Error.png
>
>
> My dag has three SparkSubmitOperators and throws this error
>
> {{```[2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,547] \{base_hook.py:84} INFO - Using connection to: id:
> spark_default. Host: yarn, Port: None, Schema: None, Login: None, Password:
> None, extra: \{'deploy-mode': 'cluster', 'spark-binary':
> '/usr/bin/spark2-submit/'}
> [2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,548] \{spark_submit_hook.py:302} INFO - Spark-Submit cmd:
> ['spark-submit', '--master', 'yarn', '--conf', 'spark.yarn.maxAppAttempts=2',
> '--files', '/home/centos/airflow/dags/conf/aws_keys.pickle',
> '--num-executors', '4', '--executor-cores', '4', '--executor-memory', '20G',
> '--name', 'universal_cross_matrix_joins_AUS_27-11-2019', '--deploy-mode',
> 'cluster',
> '/home/centos/airflow/scripts/near-airflow-repo/scripts/universal_cross_matrix/join_ifa_ncid.py',
> '--input_date', '27-11-2019', '--country', 'AUS', '--tenant_id', '63ce7457']
> [2019-11-29 12:58:54,553] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,553] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,555] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,555] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,557] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,557] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,560] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,560] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,563] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,563] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,565] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,564] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,567] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,567] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,569] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,569] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,571] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,571] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,574] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,574] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,576] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,576] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory```}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
