ajaymalikbol opened a new issue #22269: URL: https://github.com/apache/airflow/issues/22269
### Apache Airflow version 2.2.2 ### What happened We recently upgraded the airflow version from 2.0.2 to 2.2.2 In one of the dag, we are getting the issue in the python operator. Error: Task exited with return code Negsignal.SIGABRT The dag consists of multiple tasks and out of those tasks one particular task (python operator) is showing the above error consistently. we are seeing the log Started process 535 to run the task and then it is failing with the above error. We are also seeing file [core.535 getting generated in the disk](https://www.siteground.com/kb/core-xxxxx-files/) which indicates that there is some issue with the application. Though it is working fine in local but failing in AWS ECS cluster ``` *** Reading local file: /opt/airflow/logs/HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0/update_redshift_table_for_cluster_index_10_0/2022-03-14T08:01:00+00:00/1.log [2022-03-14, 09:05:15 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0.update_redshift_table_for_cluster_index_10_0 scheduled__2022-03-14T08:01:00+00:00 [queued]> [2022-03-14, 09:05:15 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0.update_redshift_table_for_cluster_index_10_0 scheduled__2022-03-14T08:01:00+00:00 [queued]> [2022-03-14, 09:05:15 UTC] {taskinstance.py:1241} INFO - -------------------------------------------------------------------------------- [2022-03-14, 09:05:15 UTC] {taskinstance.py:1242} INFO - Starting attempt 1 of 1 [2022-03-14, 09:05:15 UTC] {taskinstance.py:1243} INFO - -------------------------------------------------------------------------------- [2022-03-14, 09:05:15 UTC] {taskinstance.py:1262} INFO - Executing <Task(PythonOperator): update_redshift_table_for_cluster_index_10_0> on 2022-03-14 08:01:00+00:00 [2022-03-14, 09:05:15 UTC] {standard_task_runner.py:52} INFO - Started process 535 to run task [2022-03-14, 09:05:15 UTC] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0', 'update_redshift_table_for_cluster_index_10_0', 'scheduled__2022-03-14T08:01:00+00:00', '--job-id', '11', '--raw', '--subdir', 'DAGS_FOLDER/hoodie_etl_pipeline_scheduler.py', '--cfg-path', '/tmp/tmpaqj_dwlc', '--error-file', '/tmp/tmpe9dznvdr'] [2022-03-14, 09:05:15 UTC] {standard_task_runner.py:77} INFO - Job 11: Subtask update_redshift_table_for_cluster_index_10_0 [2022-03-14, 09:05:15 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0.update_redshift_table_for_cluster_index_10_0 scheduled__2022-03-14T08:01:00+00:00 [running]> on host 66bd98e63fe3 [2022-03-14, 09:05:16 UTC] {taskinstance.py:1427} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0 AIRFLOW_CTX_TASK_ID=update_redshift_table_for_cluster_index_10_0 AIRFLOW_CTX_EXECUTION_DATE=2022-03-14T08:01:00+00:00 AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-03-14T08:01:00+00:00 [2022-03-14, 09:05:16 UTC] {logging_mixin.py:109} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py:1941 DeprecationWarning: Accessing 'execution_date' from the template is deprecated and will be removed in a future version. Please use 'logical_date' or 'data_interval_start' instead. [2022-03-14, 09:05:16 UTC] {logging_mixin.py:109} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py:1941 DeprecationWarning: Accessing 'next_ds' from the template is deprecated and will be removed in a future version. Please use 'data_interval_end | ds' instead. [2022-03-14, 09:05:17 UTC] {local_task_job.py:154} INFO - Task exited with return code Negsignal.SIGABRT [2022-03-14, 09:05:17 UTC] {taskinstance.py:1270} INFO - Marking task as FAILED. dag_id=HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0, task_id=update_redshift_table_for_cluster_index_10_0, execution_date=20220314T080100, start_date=20220314T090515, end_date=20220314T090517 [2022-03-14, 09:05:17 UTC] {local_task_job.py:264} INFO - 1 downstream tasks scheduled from follow-on schedule check ### What you think should happen instead Python operator should have worked ### How to reproduce _No response_ ### Operating System PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/"; SUPPORT_URL="https://www.debian.org/support"; BUG_REPORT_URL="https://bugs.debian.org/"; ### Versions of Apache Airflow Providers _No response_ ### Deployment Docker-Compose ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
