pankajkoti commented on PR #32438: URL: https://github.com/apache/airflow/pull/32438#issuecomment-1627194927
hi @Owen-CH-Leung . Please find below steps I used to set up Filebeat to ship logs. @jedcunningham helped me to set that up, so thanks a ton to him. I am using Docker Desktop's Kubernetes cluster on MacOS, so you might unfortunately need Docker Desktop's Kubernetes cluster for the below steps to work. I have attached 4 files to be used, please remove the `.txt` suffix from each of the files after downloading(had to append `.txt` as GitHub does not allow to append YAML files). 1. `es.yaml`: setup ES on Kubernetes `helm repo add elastic https://helm.elastic.co` `helm repo update` `helm upgrade --install -f es.yaml pk-elasticsearch elastic/elasticsearch` `kubectl port-forward svc/elasticsearch-master 9200:9200` 2. `kibana.yaml`: Kibana on Kubernetes for UI to verify logs are getting shipped to ES `helm upgrade --install -f kibana.yaml pk-kibana elastic/kibana --version 7.17.3 ` `kubectl port-forward svc/pk-kibana-kibana 5601:5601` 3. `filebeat_pv.yaml` : For volumes required for Filebeat In this file, please replace the `hostpath -> path` key's value to your local airflow repo path's logs directory. When you use breeze, the logs will be available in the `<your_clone_repo_path>/airflow/logs` directory and Filebeat will ship logs from this directory to ES then. `kubectl apply -f filebeat_pv.yaml` 4. `filebeat.yaml` : Filebeat on Kubernetes `helm upgrade --install -f filebeat.yaml pk-filebeat elastic/filebeat` There might be a slight delay in shipping the logs but you will see them eventually in a minute using Kibana UI's Logs feature. [es.yaml.txt](https://github.com/apache/airflow/files/11990542/es.yaml.txt) [filebeat_pv.yaml.txt](https://github.com/apache/airflow/files/11990543/filebeat_pv.yaml.txt) [filebeat_pv.yaml.txt](https://github.com/apache/airflow/files/11990556/filebeat_pv.yaml.txt) [filebneat.yaml.txt](https://github.com/apache/airflow/files/11990557/filebneat.yaml.txt) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
