GitHub user frodo2000 created a discussion: SSL: CERTIFICATE_VERIFY_FAILED in
airflow-worker for certificates with internal CA
### Apache Airflow version
3.0.6
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When API server has setup SSL certificate with local Certificate Authority,
airflow-worker is not enable to finish task and service log contains the
following error:
`Sep 11 13:13:06 dwh-airflow-dev bash[10802]: [2025-09-11 13:13:06 +0000]
[10802] [INFO] Handling signal: term
Sep 11 13:13:06 dwh-airflow-dev bash[10804]: [2025-09-11 13:13:06 +0000]
[10804] [INFO] Worker exiting (pid: 10804)
...skipping...
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ )
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ do =
<tenacity.DoAttempt object at 0x7dc38a77b2f0>
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ kwargs = {
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'content':
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
'{"state":"running","hostname":"dwh-airflow-dev.xxx.xx","unixname":"airflo… │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'data':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'files':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'json':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'params':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'headers':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'cookies':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'auth':
<httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
'follow_redirects': <httpx._client.UseClientDefault object at │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 0x7dc398afe090>,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'timeout':
<httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ ... +1
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ }
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ retry_state = <RetryCallState
138278795187296: attempt #5; slept for 6.92; last result: │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ failed
(ConnectError [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ failed: unable
to get local issuer certificate (_ssl.c:1000))> │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ self = <Retrying object
at 0x7dc390fe3ad0 (stop=<tenacity.stop.stop_after_attempt │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ object at
0x7dc38a743020>, wait=<retryhttp._wait.wait_context_aware object at │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 0x7dc38a742ff0>,
sleep=<function sleep at 0x7dc38a8e6b60>, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
retry=<tenacity.retry.retry_any object at 0x7dc38a742fc0>, before=<function
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ before_nothing
at 0x7dc38a8e7d80>, after=<function after_nothing at │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
0x7dc38a8e6ac0>)>
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/airflow/sdk/api/client.py:735
in request │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ ❱ 735 return
super().request(*args, **kwargs) │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
╭─────────────────────────────────────────── locals
───────────────────────────────────────────╮ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ args = ('PATCH',
'task-instances/01993cd3-9124-7f15-9d8a-9f86b86d6c2d/run') │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ kwargs = {
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'content':
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
'{"state":"running","hostname":"dwh-airflow-dev.xxx.xx","unixname":"airflow","'…
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'data': None,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'files': None,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'json': None,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'params': None,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'headers': None,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'cookies': None,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'auth':
<httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
'follow_redirects': <httpx._client.UseClientDefault object at 0x7dc398afe090>,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'timeout':
<httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ ... +1
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ }
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ self =
<airflow.sdk.api.client.Client object at 0x7dc38a8517c0>
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:825 in
request │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:914 in
send │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:942 in
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ _send_handling_auth
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:979 in
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ _send_handling_redirects
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:1014
in │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ _send_single_request
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_transports/default.py:249
in │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ handle_request
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/usr/lib/python3.12/contextlib.py:158 in __exit__
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_transports/default.py:118
in │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ map_httpcore_exceptions
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]:
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: ConnectError: [SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: certificate (_ssl.c:1000)
`
In SDK Client class I found that only server certificate is added to Certifi
library:
` ctx = ssl.create_default_context(cafile=certifi.where())
if API_SSL_CERT_PATH:
ctx.load_verify_locations(API_SSL_CERT_PATH)`
Then only self-signed certificates are valid but certificates with local CA (or
even intermediate local CA) becomes invalid.
We should consider add additional config parameter like API_SSL_CA_BUNDLE_PATH
and add it to certifi context.
Checked workaround:
- adding CA certificated directly to certifi/cacert.pem file - airflow-worker
works correctly but each certifi update needs to cacert.pem redefinition
- adding env variables REQUESTS_CA_BUNDLE or SSL_CERT_FILE doesn't work
### What you think should happen instead?
_No response_
### How to reproduce
Create SSL certificate with local CA chain
### Operating System
Ubuntu
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
Systemd services for:
- postgresql
- redis
- airflow-api
- ariflow-scheduler
- airflow-dag-processor
- airflow-triggerer
- airflow-worker
with env configuration in /etc/airflow.cfg file
Celery Executor used
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
GitHub link: https://github.com/apache/airflow/discussions/56590
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
