jhgoebbert commented on PR #53821: URL: https://github.com/apache/airflow/pull/53821#issuecomment-3992821849
Thank you @Owen-CH-Leung I have tested the new feature with the CeleryExecutor and KubernetesExecutor in combination with https://github.com/apache/airflow/pull/62121 and it works great! Fantastic!! I can read from the source of [ElasticsearchRemoteLogIO(LoggingMixin).upload()](https://github.com/apache/airflow/blob/main/providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py#L557) that its [self._write_to_es()](https://github.com/apache/airflow/blob/main/providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py#L605) directly tries to communicate with the ElasticSearch server through [helpers.bulk()](https://github.com/apache/airflow/blob/main/providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py#L634) Am I correct that at the moment the ElasticSearch server must be reachable directly from the task's location. The logs are not send using the TaskSDK - so the EdgeExecutor might fail from other clusters if the ElasticSearch server is unreachable from there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
