frodo2000 opened a new issue, #55557:
URL: https://github.com/apache/airflow/issues/55557
### Apache Airflow version
3.0.6
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When API server has setup SSL certificate with local Certificate Authority,
airflow-worker is not enable to finish task and service log contains the
following error:
`Sep 11 13:13:06 dwh-airflow-dev bash[10802]: [2025-09-11 13:13:06 +0000]
[10802] [INFO] Handling signal: term
Sep 11 13:13:06 dwh-airflow-dev bash[10804]: [2025-09-11 13:13:06 +0000]
[10804] [INFO] Worker exiting (pid: 10804)
...skipping...
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ )
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ do =
<tenacity.DoAttempt object at 0x7dc38a77b2f0>
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ kwargs = {
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
'content': │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
'{"state":"running","hostname":"dwh-airflow-dev.xxx.xx","unixname":"airflo… │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'data':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'files':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'json':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'params':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
'headers': None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
'cookies': None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'auth':
<httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
'follow_redirects': <httpx._client.UseClientDefault object at │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
0x7dc398afe090>,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
'timeout': <httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ ... +1
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ }
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ retry_state =
<RetryCallState 138278795187296: attempt #5; slept for 6.92; last result:
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ failed
(ConnectError [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ failed:
unable to get local issuer certificate (_ssl.c:1000))> │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ self = <Retrying
object at 0x7dc390fe3ad0 (stop=<tenacity.stop.stop_after_attempt │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ object at
0x7dc38a743020>, wait=<retryhttp._wait.wait_context_aware object at │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
0x7dc38a742ff0>, sleep=<function sleep at 0x7dc38a8e6b60>,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
retry=<tenacity.retry.retry_any object at 0x7dc38a742fc0>, before=<function
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
before_nothing at 0x7dc38a8e7d80>, after=<function after_nothing at
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
0x7dc38a8e6ac0>)>
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/airflow/sdk/api/client.py:735
in request │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ ❱ 735 return
super().request(*args, **kwargs) │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
╭─────────────────────────────────────────── locals
───────────────────────────────────────────╮ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ args = ('PATCH',
'task-instances/01993cd3-9124-7f15-9d8a-9f86b86d6c2d/run') │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ kwargs = {
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'content':
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
'{"state":"running","hostname":"dwh-airflow-dev.xxx.xx","unixname":"airflow","'…
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'data': None,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'files': None,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'json': None,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'params':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'headers':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'cookies':
None, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'auth':
<httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │
'follow_redirects': <httpx._client.UseClientDefault object at 0x7dc398afe090>,
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'timeout':
<httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ ... +1
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ }
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ self =
<airflow.sdk.api.client.Client object at 0x7dc38a8517c0>
│ │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:825 in
request │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:914 in
send │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:942 in
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ _send_handling_auth
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:979 in
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ _send_handling_redirects
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:1014
in │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ _send_single_request
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_transports/default.py:249
in │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ handle_request
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/usr/lib/python3.12/contextlib.py:158 in __exit__
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │
/home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_transports/default.py:118
in │
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ map_httpcore_exceptions
│
Sep 12 07:28:54 dwh-airflow-dev bash[27220]:
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: ConnectError: [SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer
Sep 12 07:28:54 dwh-airflow-dev bash[27220]: certificate (_ssl.c:1000)
`
In SDK Client class I found that only server certificate is added to Certifi
library:
` ctx = ssl.create_default_context(cafile=certifi.where())
if API_SSL_CERT_PATH:
ctx.load_verify_locations(API_SSL_CERT_PATH)`
Then only self-signed certificates are valid but certificates with local CA
(or even intermediate local CA) becomes invalid.
We should consider add additional config parameter like
API_SSL_CA_BUNDLE_PATH and add it to certifi context.
Checked workaround:
- adding CA certificated directly to certifi/cacert.pem file -
airflow-worker works correctly but each certifi update needs to cacert.pem
redefinition
- adding env variables REQUESTS_CA_BUNDLE or SSL_CERT_FILE doesn't work
### What you think should happen instead?
_No response_
### How to reproduce
Create SSL certificate with local CA chain
### Operating System
Ubuntu
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
Systemd services for:
- postgresql
- redis
- airflow-api
- ariflow-scheduler
- airflow-dag-processor
- airflow-triggerer
- airflow-worker
with env configuration in /etc/airflow.cfg file
Celery Executor used
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
