andormarkus commented on issue #13824: URL: https://github.com/apache/airflow/issues/13824#issuecomment-1057844236
Hello, We are running our Airflow on AWS EKS with the latest helm chart and official docker image. I don't think it is setup issue. I was able to reproduce it locally as well with docker compose: 1. Download the official docker compose file `curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.2.4/docker-compose.yaml` . 2. Create a test CloudWatch log group. 3. Create `dags` directory with `simple_dag.py` 5 times. 4. Edit the `x-airflow-common.environment` section in the `docker-compose.yaml` file the following way: Disable example dags ```yaml AIRFLOW__CORE__LOAD_EXAMPLES: 'false' ``` Add the following environment variables. Adjust your `region` according to your preferences. ```yaml AWS_DEFAULT_REGION: 'eu-central-1' AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN} AIRFLOW_CONN_AWS_DEFAULT: "aws://?region_name=eu-central-1" AIRFLOW__LOGGING__REMOTE_LOGGING: "True" AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: AWS_DEFAULT AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: cloudwatch://<cloudwatch-log-group-arn> ``` 5. Insert you AWS environment variables into terminal. (we are using AWS SSO to login etc...) ```shell export AWS_ACCESS_KEY_ID="########" export AWS_SECRET_ACCESS_KEY="########" export AWS_SESSION_TOKEN="########" ``` 6. Setting the right Airflow user: `echo -e "AIRFLOW_UID=$(id -u)" > .env` 7. Initiate docker compose environment: `docker compose up -d` 8. Login to webui <http://localhost:8080/> and enable dags.  9. Check the worker logs after minimum 60 minutes. **It only happens only 1-2% of the dag runs.** ```shell ▶ docker logs airflow_dag_test-airflow-worker-1 --since=60m 2>&1 | grep ERROR [2022-03-03 08:34:21,817: ERROR/ForkPoolWorker-15] Failed to execute task Task received SIGTERM signal. [2022-03-03 08:38:22,608: ERROR/ForkPoolWorker-15] Failed to execute task Task received SIGTERM signal. [2022-03-03 09:01:21,553: ERROR/ForkPoolWorker-16] Failed to execute task Task received SIGTERM signal. [2022-03-03 09:11:21,887: ERROR/ForkPoolWorker-3] Failed to execute task Task received SIGTERM signal. [2022-03-03 09:11:21,898: ERROR/ForkPoolWorker-2] Failed to execute task Task received SIGTERM signal. [2022-03-03 09:15:21,876: ERROR/ForkPoolWorker-15] Failed to execute task Task received SIGTERM signal. [2022-03-03 09:15:21,876: ERROR/ForkPoolWorker-1] Failed to execute task Task received SIGTERM signal. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
