Owen-CH-Leung opened a new pull request, #41799:
URL: https://github.com/apache/airflow/pull/41799
Closes #33619
Airflow currently only supports reading remote log from elasticsearch. This
PR adds feature to also allow reading remote logs from opensearch.
Similar to reading remote logs from elasticsearch, users need to use other
tools to import logs into Opensearch.
The set up is very similar to setting up remote logging for Elasticsearch.
In the airflow.cfg:
```
remote_logging = True
remote_log_conn_id = opensearch_default
[opensearch]
host = [OS host name] (e.g. my-opensearch-01)
port = [port number] (e.g. 9200)
username =
password =
```
Depending on your Opensearch config, you may also add the following in
airflow.cfg :
```
[opensearch_configs]
http_compress = False
use_ssl = False
verify_certs = False
ssl_assert_hostname = False
ssl_show_warn = False
ca_certs =
```
If you set up everything successfully, webserver should output logs like the
following:
```
[2024-08-27T15:02:54.611+0000] {base.py:258} INFO - POST
http://my-opensearch-01:9200/_all/_count [status:200 request:0.034s]
[2024-08-27T15:02:54.654+0000] {base.py:258} INFO - POST
http://my-opensearch-01:9200/_all/_search?size=1000&sort=log&from=0 [status:200
request:0.040s]
192.168.65.1 - - [27/Aug/2024:15:02:54 +0000] "GET
/api/v1/dags/consume_1_or_2_with_dataset_expressions/dagRuns/manual__2024-08-27T15:00:00.024040+00:00/taskInstances/consume_1_or_2_with_dataset_expressions/logs/1?full_content=false
HTTP/1.1" 200 4219
"http://localhost:28080/dags/consume_1_or_2_with_dataset_expressions/grid?dag_run_id=manual__2024-08-27T15%3A00%3A00.024040%2B00%3A00&task_id=consume_1_or_2_with_dataset_expressions&tab=logs";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
