GlobalKeeper opened a new issue #17463: URL: https://github.com/apache/airflow/issues/17463
**Apache Airflow version**: v2.1.2 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): none **Environment**: - **Cloud provider or hardware configuration**: self-hosted airflow (with scheduler) started in docker, dind container linked to airflows containers, sentry on-premise - **OS** (e.g. from /etc/os-release): ubuntu 20.04 - **Kernel** (e.g. `uname -a`): 5.4.0-67-generic - **Install tools**: - **Others**: Docker version 20.10.3, build 48d30b5 **What happened**: Sometime notify exception: NotFound 404 Client Error for http://airflow-dind:2375/v1.41/containers/<Container_ID>/stop: Not Found ("No such container: <Сontainer_ID>") <Container_ID> = any random string **What you expected to happen**: No events in sentry (exceptions) **How to reproduce it**: ``` DAG from datetime import datetime from airflow import DAG from airflow.providers.docker.operators.docker import DockerOperator from airflow.models import Variable kwargs = { 'schedule_interval': '*/5 * * * *', 'start_date': datetime(2021, 7, 2), 'description': 'none', 'catchup': False, 'max_active_runs': 1 } with DAG('test', **kwargs) as dag: DockerOperator( task_id='test', docker_url='http://airflow-dind:2375', api_version='auto', image='debian:10', environment=dag_config, command='sh -c "exit 1"', network_mode='host', auto_remove=True, force_pull=True ) ``` **Anything else we need to know**: docker-compose.example: ``` version: '3' x-airflow-common: &airflow-common image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.2} environment: &airflow-common-env AIRFLOW__CORE__EXECUTOR: LocalExecutor AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow AIRFLOW__CORE__FERNET_KEY: '' AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true' AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth' _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-} volumes: - ./dags:/opt/airflow/dags - ./logs:/opt/airflow/logs - ./plugins:/opt/airflow/plugins - /tmp:/tmp depends_on: postgres: services: postgres: image: postgres:13 environment: POSTGRES_USER: airflow POSTGRES_PASSWORD: airflow POSTGRES_DB: airflow volumes: - postgres-db-volume:/var/lib/postgresql/data airflow-webserver: <<: *airflow-common command: webserver ports: - 8080:8080 airflow-scheduler: <<: *airflow-common command: scheduler airflow-dind: image: docker:dind environment: DOCKER_TLS_CERTDIR: "" volumes: - /tmp:/tmp volumes: postgres-db-volume: ``` How often does this problem occur? Once? Every time etc? 5-10 times per 288 jobs Any relevant logs to include? Put them here in side a detail tag: ``` .... [2021-08-06 11:45:13,266] {local_task_job.py:198} WARNING - State of this instance has been externally set to success. Terminating instance. [2021-08-06 11:45:13,268] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 67244 [2021-08-06 11:45:13,284] {taskinstance.py:1284} ERROR - Received SIGTERM. Terminating subprocesses. [2021-08-06 11:45:13,285] {docker.py:345} INFO - Stopping docker container [2021-08-06 11:45:13,692] {poolmanager.py:375} INFO - Redirecting http://sentry.ru/api/24/store/ -> https://sentry.ru/api/24/store/ [2021-08-06 11:45:13,764] {process_utils.py:66} INFO - Process psutil.Process(pid=67244, status='terminated', exitcode=1, started='11:45:01') (67244) terminated with exit code 1 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
