xisberto opened a new issue, #39225:
URL: https://github.com/apache/airflow/issues/39225

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.8.3
   
   ### What happened?
   
   I have Airflow 2.8.3 on Kubernetes, deployed through the Helm chart. On the 
same namespace, I have a MinIO tenant.
   
   My DAGs can write output files to the MinIO using a connection and the 
S3Hook.
   
   I'm trying to use remote_logging to get the logs to a bucket on the same 
MinIO tenant, but it's not working.
   
   When I try to read the DAGs logs, the airflow-webserver logs say:
   
   ```
   [2024-04-23T16:34:41.070+0000] {base.py:83} INFO - Using connection ID 
'conn_logs' for task execution.
   [2024-04-23T16:34:41.072+0000] {connection_wrapper.py:387} INFO - AWS 
Connection (conn_id='conn_logs', conn_type='aws') credentials retrieved from 
extra.
   
/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py:1061 
InsecureRequestWarning: Unverified HTTPS request is being made to host 
'airflow-storage-hl'. Adding certificate verification is strongly advised. See: 
https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
   [2024-04-23T16:34:42.721+0000] {file_task_handler.py:583} ERROR - Could not 
read served logs
   Traceback (most recent call last):
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_exceptions.py", 
line 10, in map_exceptions
       yield
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/backends/sync.py", 
line 94, in connect_tcp
       sock = socket.create_connection(
     File "/usr/local/lib/python3.8/socket.py", line 787, in create_connection
       for res in getaddrinfo(host, port, 0, SOCK_STREAM):
     File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
       for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
   socket.gaierror: [Errno -3] Temporary failure in name resolution
   During handling of the above exception, another exception occurred:
   Traceback (most recent call last):
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpx/_transports/default.py",
 line 60, in map_httpcore_exceptions
       yield
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpx/_transports/default.py",
 line 218, in handle_request
       resp = self._pool.handle_request(req)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py",
 line 253, in handle_request
       raise exc
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py",
 line 237, in handle_request
       response = connection.handle_request(request)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection.py",
 line 86, in handle_request
       raise exc
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection.py",
 line 63, in handle_request
       stream = self._connect(request)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection.py",
 line 111, in _connect
       stream = self._network_backend.connect_tcp(**kwargs)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/backends/sync.py", 
line 94, in connect_tcp
       sock = socket.create_connection(
     File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
       self.gen.throw(type, value, traceback)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpcore/_exceptions.py", 
line 14, in map_exceptions
       raise to_exc(exc)
   httpcore.ConnectError: [Errno -3] Temporary failure in name resolution
   The above exception was the direct cause of the following exception:
   Traceback (most recent call last):
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py",
 line 561, in _read_from_logs_server
       response = _fetch_logs_from_service(url, rel_path)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py",
 line 92, in _fetch_logs_from_service
       response = httpx.get(
     File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_api.py", 
line 189, in get
       return request(
     File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_api.py", 
line 100, in request
       return client.request(
     File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py", 
line 821, in request
       return self.send(request, auth=auth, follow_redirects=follow_redirects)
     File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py", 
line 908, in send
       response = self._send_handling_auth(
     File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py", 
line 936, in _send_handling_auth
       response = self._send_handling_redirects(
     File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py", 
line 973, in _send_handling_redirects
       response = self._send_single_request(request)
     File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py", 
line 1009, in _send_single_request
       response = transport.handle_request(request)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpx/_transports/default.py",
 line 218, in handle_request
       resp = self._pool.handle_request(req)
     File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
       self.gen.throw(type, value, traceback)
     File 
"/home/airflow/.local/lib/python3.8/site-packages/httpx/_transports/default.py",
 line 77, in map_httpcore_exceptions
       raise mapped_exc(message) from exc
   httpx.ConnectError: [Errno -3] Temporary failure in name resolution
   10.244.250.0 - - [23/Apr/2024:16:34:42 +0000] "GET 
/api/v1/dags/extract-entradas-saidas/dagRuns/manual__2024-04-23T13:01:22.569932+00:00/taskInstances/extract_and_combine_task/logs/2?full_content=false
 HTTP/1.1" 200 286 
"https://airflow.mpce.mp.br/dags/extract-entradas-saidas/grid?dag_run_id=manual__2024-04-23T13%3A01%3A22.569932%2B00%3A00&task_id=extract_and_combine_task&tab=logs";
 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/124.0.0.0 Safari/537.36"
   ```
   
   This is the relevant part of the values.yaml:
   
   ```
   config:
     logging:
       remote_logging: 'True'
       logging_level: 'INFO'
       remote_base_log_folder: 's3://logs'
       remote_log_conn_id: 'conn_logs'
       delete_worker_pods: 'False'
       encrypt_s3_logs: 'False'
       colored_console_log: 'True'
   ```
   
   This is the `conn_logs` connection extras currently:
   
   ```
   {
     "aws_access_key_id": "XXXXXXXXXXXXXX",
     "aws_secret_access_key": "XXXXXXXXXXXXXXXXXXXXX",
     "endpoint_url": "https://airflow-storage-hl:9000";,
     "verify": false
   }
   ```
   
   What I've tried to do:
   
   - Use the complete URL for the `endpoint_url` as in 
`https://airflow-storage-hl.airflow.svc.cluster.local:9000`
   - Use the cluster's CA file as the `verify` value 
(`/var/run/secrets/kubernetes.io/serviceaccount/ca.crt`)
   
   Before setting the `verify` option in the connection, I was getting errors 
related to the certificate validation. This also was the error when setting the 
value to `'False'`.
   
   ### What you think should happen instead?
   
   The logs should be written to the MinIO 'logs' bucket the same way the DAGs 
write files to the other buckets in the same MinIO tenant.
   
   ### How to reproduce
   
   1. Deploy MinIO via helm chart
   2. Create a MinIO tenant `airflow-storage`. Create a Access Key to use in 
the Airflow Connection
   3. Deploy Airflow via helm chart, configure the Airflow Connection
   4. Configure remote logging in the Airflow's values.yaml file, redeploy it
   
   ### Operating System
   
   Rocky Linux 9.3
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-amazon==8.19.0
   apache-airflow-providers-celery==3.6.1
   apache-airflow-providers-cncf-kubernetes==8.0.1
   apache-airflow-providers-common-io==1.3.0
   apache-airflow-providers-common-sql==1.11.1
   apache-airflow-providers-docker==3.9.2
   apache-airflow-providers-elasticsearch==5.3.3
   apache-airflow-providers-ftp==3.7.0
   apache-airflow-providers-google==10.16.0
   apache-airflow-providers-grpc==3.4.1
   apache-airflow-providers-hashicorp==3.6.4
   apache-airflow-providers-http==4.10.0
   apache-airflow-providers-imap==3.5.0
   apache-airflow-providers-microsoft-azure==9.0.1
   apache-airflow-providers-microsoft-mssql==3.5.0
   apache-airflow-providers-mysql==5.5.4
   apache-airflow-providers-odbc==4.4.1
   apache-airflow-providers-openlineage==1.6.0
   apache-airflow-providers-postgres==5.10.2
   apache-airflow-providers-redis==3.6.0
   apache-airflow-providers-sendgrid==3.4.0
   apache-airflow-providers-sftp==4.9.0
   apache-airflow-providers-slack==8.6.1
   apache-airflow-providers-smtp==1.6.1
   apache-airflow-providers-snowflake==5.3.1
   apache-airflow-providers-sqlite==3.7.1
   apache-airflow-providers-ssh==3.10.1
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   I'm deploying Airflow using the official Helm Chart using ArgoCD. The image 
is modified as follows:
   
   ```
   FROM apache/airflow:2.8.3
   RUN pip install "apache-airflow==${AIRFLOW_VERSION}" --no-cache-dir pip 
install apache-airflow-providers-microsoft-mssql==3.5.0
   ```
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to