brunocmacedo commented on issue #59707: URL: https://github.com/apache/airflow/issues/59707#issuecomment-3687397662
After a couple hours trying to find what was the issue after 3.1.5 upgrade I found this and can confirm that I have the same issue with [apache-airflow-providers-celery 3.14.1](https://pypi.org/project/apache-airflow-providers-celery/). Falling back to 3.13.1 solved the issue for now. As a reference, here is the airflow-worker definition in my docker compose. In my case i need to set celery-hostname due to the three replicas being used. ```yaml airflow-worker: <<: *airflow-common # container name is not set due to the usage of replicas #container_name: airflow-worker command: celery worker --celery-hostname=%h healthcheck: test: - "CMD-SHELL" - 'celery --app airflow.providers.celery.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}" || celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"' interval: 30s timeout: 10s retries: 5 start_period: 30s environment: <<: *airflow-common-env # Required to handle warm shutdown of the celery workers properly # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation DUMB_INIT_SETSID: "0" restart: always depends_on: <<: *airflow-common-depends-on airflow-apiserver: condition: service_healthy airflow-init: condition: service_completed_successfully # below is not set so flower can detect the different workers #hostname: "{{.Node.ID}}-{{.Task.Slot}}" deploy: replicas: 3 # Number of worker replicas to run, increase this to create multiple worker containers allowing more parallel task execution mem_swappiness: 0 # Disable swap to prevent slowdowns ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
