ajaymalikbol opened a new issue #19635: URL: https://github.com/apache/airflow/issues/19635
### Apache Airflow version 2.0.2 ### Operating System PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/"; SUPPORT_URL="https://www.debian.org/support"; BUG_REPORT_URL="https://bugs.debian.org/"; ### Versions of Apache Airflow Providers apache-airflow-backport-providers-google==2021.3.3 apache-airflow-backport-providers-amazon==2021.3.3 ### Deployment Docker-Compose ### Deployment details _No response_ ### What happened We are using airflow to schedule ETL pipeline and for the same transformation we are using EMR step sensor. And we are seeing `Received SIGTERM. Terminating subprocesses.` error very frequently. [https://github.com/apache/airflow/blob/88583095c408ef9ea60f793e7072e3fd4b88e329/airflow/models/taskinstance.py#L1394](url) From logs it looks like Airflow task is setting the operator status = Success and again airflow is considering this change invalid `State of this instance has been externally set to success. Terminating instance.` [https://github.com/apache/airflow/blob/88583095c408ef9ea60f793e7072e3fd4b88e329/airflow/jobs/local_task_job.py#L211](url) From Airflow UI there is no Failures but we are seeing the below errors in the airflow logs. ```[2021-11-08 01:12:13,303] {taskinstance.py:1089} INFO - Executing <Task(EmrStepSensor): etl_step_sensor_task> on 2021-11-08T00:01:00+00:00 [2021-11-08 01:12:13,405] {standard_task_runner.py:52} INFO - Started process 27548 to run task [2021-11-08 01:12:14,178] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'ETL_PIPELINE_JOB', 'etl_step_sensor_task', '2021-11-08T00:01:00+00:00', '--job-id', '100495', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/etl_pipeline_script.py', '--cfg-path', '/tmp/tmp_6kklqa5', '--error-file', '/tmp/tmpo2ug1xs3'] [2021-11-08 01:12:14,279] {standard_task_runner.py:77} INFO - Job 100495: Subtask etl_step_sensor_task [2021-11-08 01:12:15,800] {logging_mixin.py:104} INFO - Running <TaskInstance: ETL_PIPELINE_JOB.etl_step_sensor_task 2021-11-08T00:01:00+00:00 [running]> on host e363e366c87b [2021-11-08 01:12:17,015] {taskinstance.py:1281} INFO - Exporting the following env vars: [email protected] AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=ETL_PIPELINE_JOB AIRFLOW_CTX_TASK_ID=etl_step_sensor_task AIRFLOW_CTX_EXECUTION_DATE=2021-11-08T00:01:00+00:00 AIRFLOW_CTX_DAG_RUN_ID=scheduled__2021-11-08T00:01:00+00:00 [2021-11-08 01:12:17,174] {base_aws.py:362} INFO - Airflow Connection: aws_conn_id=aws_default [2021-11-08 01:12:17,462] {base_aws.py:385} WARNING - Unable to use Airflow Connection for credentials. [2021-11-08 01:12:17,462] {base_aws.py:386} INFO - Fallback on boto3 credential strategy [2021-11-08 01:12:17,462] {base_aws.py:389} INFO - Creating session using boto3 credential strategy region_name=None [2021-11-08 01:12:18,035] {emr_step.py:75} INFO - Poking step step-id on cluster cluster_id [2021-11-08 01:12:18,395] {emr_base.py:68} INFO - Job flow currently COMPLETED [2021-11-08 01:12:18,395] {base.py:245} INFO - Success criteria met. Exiting. [2021-11-08 01:12:18,522] {taskinstance.py:1185} INFO - Marking task as SUCCESS. dag_id=ETL_PIPELINE_JOB, task_id=etl_step_sensor_task, execution_date=20211108T000100, start_date=20211108T011212, end_date=20211108T011218 [2021-11-08 01:12:19,269] {local_task_job.py:187} WARNING - State of this instance has been externally set to success. Terminating instance. [2021-11-08 01:12:19,272] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 27548 [2021-11-08 01:12:19,419] {taskinstance.py:1265} ERROR - Received SIGTERM. Terminating subprocesses. [2021-11-08 01:12:20,636] {process_utils.py:66} INFO - Process psutil.Process(pid=27548, status='terminated', exitcode=1, started='01:12:12') (27548) terminated with exit code 1 airflow@e363e366c87b:/opt/airflow/logs/ETL_PIPELINE_JOB$ ### What you expected to happen We expect No errors in the airflow task ``` Received SIGTERM. Terminating subprocesses. And if this expected, How we can fix it in Airflow Build. ### How to reproduce This is an intermittent issue, Not sure if we can generate these errors with some pre-defined steps ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
