xisberto opened a new issue, #39225:
URL: https://github.com/apache/airflow/issues/39225
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.8.3
### What happened?
I have Airflow 2.8.3 on Kubernetes, deployed through the Helm chart. On the
same namespace, I have a MinIO tenant.
My DAGs can write output files to the MinIO using a connection and the
S3Hook.
I'm trying to use remote_logging to get the logs to a bucket on the same
MinIO tenant, but it's not working.
When I try to read the DAGs logs, the airflow-webserver logs say:
```
[2024-04-23T16:34:41.070+0000] {base.py:83} INFO - Using connection ID
'conn_logs' for task execution.
[2024-04-23T16:34:41.072+0000] {connection_wrapper.py:387} INFO - AWS
Connection (conn_id='conn_logs', conn_type='aws') credentials retrieved from
extra.
/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py:1061
InsecureRequestWarning: Unverified HTTPS request is being made to host
'airflow-storage-hl'. Adding certificate verification is strongly advised. See:
https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
[2024-04-23T16:34:42.721+0000] {file_task_handler.py:583} ERROR - Could not
read served logs
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_exceptions.py",
line 10, in map_exceptions
yield
File
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/backends/sync.py",
line 94, in connect_tcp
sock = socket.create_connection(
File "/usr/local/lib/python3.8/socket.py", line 787, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/httpx/_transports/default.py",
line 60, in map_httpcore_exceptions
yield
File
"/home/airflow/.local/lib/python3.8/site-packages/httpx/_transports/default.py",
line 218, in handle_request
resp = self._pool.handle_request(req)
File
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py",
line 253, in handle_request
raise exc
File
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py",
line 237, in handle_request
response = connection.handle_request(request)
File
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection.py",
line 86, in handle_request
raise exc
File
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection.py",
line 63, in handle_request
stream = self._connect(request)
File
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection.py",
line 111, in _connect
stream = self._network_backend.connect_tcp(**kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/backends/sync.py",
line 94, in connect_tcp
sock = socket.create_connection(
File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_exceptions.py",
line 14, in map_exceptions
raise to_exc(exc)
httpcore.ConnectError: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py",
line 561, in _read_from_logs_server
response = _fetch_logs_from_service(url, rel_path)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py",
line 92, in _fetch_logs_from_service
response = httpx.get(
File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_api.py",
line 189, in get
return request(
File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_api.py",
line 100, in request
return client.request(
File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py",
line 821, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py",
line 908, in send
response = self._send_handling_auth(
File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py",
line 936, in _send_handling_auth
response = self._send_handling_redirects(
File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py",
line 973, in _send_handling_redirects
response = self._send_single_request(request)
File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py",
line 1009, in _send_single_request
response = transport.handle_request(request)
File
"/home/airflow/.local/lib/python3.8/site-packages/httpx/_transports/default.py",
line 218, in handle_request
resp = self._pool.handle_request(req)
File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File
"/home/airflow/.local/lib/python3.8/site-packages/httpx/_transports/default.py",
line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: [Errno -3] Temporary failure in name resolution
10.244.250.0 - - [23/Apr/2024:16:34:42 +0000] "GET
/api/v1/dags/extract-entradas-saidas/dagRuns/manual__2024-04-23T13:01:22.569932+00:00/taskInstances/extract_and_combine_task/logs/2?full_content=false
HTTP/1.1" 200 286
"https://airflow.mpce.mp.br/dags/extract-entradas-saidas/grid?dag_run_id=manual__2024-04-23T13%3A01%3A22.569932%2B00%3A00&task_id=extract_and_combine_task&tab=logs"
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/124.0.0.0 Safari/537.36"
```
This is the relevant part of the values.yaml:
```
config:
logging:
remote_logging: 'True'
logging_level: 'INFO'
remote_base_log_folder: 's3://logs'
remote_log_conn_id: 'conn_logs'
delete_worker_pods: 'False'
encrypt_s3_logs: 'False'
colored_console_log: 'True'
```
This is the `conn_logs` connection extras currently:
```
{
"aws_access_key_id": "XXXXXXXXXXXXXX",
"aws_secret_access_key": "XXXXXXXXXXXXXXXXXXXXX",
"endpoint_url": "https://airflow-storage-hl:9000",
"verify": false
}
```
What I've tried to do:
- Use the complete URL for the `endpoint_url` as in
`https://airflow-storage-hl.airflow.svc.cluster.local:9000`
- Use the cluster's CA file as the `verify` value
(`/var/run/secrets/kubernetes.io/serviceaccount/ca.crt`)
Before setting the `verify` option in the connection, I was getting errors
related to the certificate validation. This also was the error when setting the
value to `'False'`.
### What you think should happen instead?
The logs should be written to the MinIO 'logs' bucket the same way the DAGs
write files to the other buckets in the same MinIO tenant.
### How to reproduce
1. Deploy MinIO via helm chart
2. Create a MinIO tenant `airflow-storage`. Create a Access Key to use in
the Airflow Connection
3. Deploy Airflow via helm chart, configure the Airflow Connection
4. Configure remote logging in the Airflow's values.yaml file, redeploy it
### Operating System
Rocky Linux 9.3
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.19.0
apache-airflow-providers-celery==3.6.1
apache-airflow-providers-cncf-kubernetes==8.0.1
apache-airflow-providers-common-io==1.3.0
apache-airflow-providers-common-sql==1.11.1
apache-airflow-providers-docker==3.9.2
apache-airflow-providers-elasticsearch==5.3.3
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-google==10.16.0
apache-airflow-providers-grpc==3.4.1
apache-airflow-providers-hashicorp==3.6.4
apache-airflow-providers-http==4.10.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-microsoft-azure==9.0.1
apache-airflow-providers-microsoft-mssql==3.5.0
apache-airflow-providers-mysql==5.5.4
apache-airflow-providers-odbc==4.4.1
apache-airflow-providers-openlineage==1.6.0
apache-airflow-providers-postgres==5.10.2
apache-airflow-providers-redis==3.6.0
apache-airflow-providers-sendgrid==3.4.0
apache-airflow-providers-sftp==4.9.0
apache-airflow-providers-slack==8.6.1
apache-airflow-providers-smtp==1.6.1
apache-airflow-providers-snowflake==5.3.1
apache-airflow-providers-sqlite==3.7.1
apache-airflow-providers-ssh==3.10.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
I'm deploying Airflow using the official Helm Chart using ArgoCD. The image
is modified as follows:
```
FROM apache/airflow:2.8.3
RUN pip install "apache-airflow==${AIRFLOW_VERSION}" --no-cache-dir pip
install apache-airflow-providers-microsoft-mssql==3.5.0
```
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]