Hi all, While reviewing “Fix the write-to-es feature #53821” [1], I found that the ElasticsearchTaskHandler (and OpensearchTaskHandler as well) has been broken for a long time since Airflow 3, as we block the DB session, but ElasticsearchTaskHandler still depends on the LogTemplate [2] db model for the log writing path [3]. To fix the ElasticsearchTaskHandler in the Airflow 3 security model, we can either:
1. Drop LogTemplate db model support for ElasticsearchTaskHandler, or 2. Still support the LogTemplate feature with ElasticsearchTaskHandler by fetching the LogTemplate db model before we block the DB session and attaching the LogTemplate attribute to the ExecuteTask workload. For 1. It will be much simpler and will also decouple the ES provider from Airflow Core, in my opinion, but with no backward compatibility (we need to introduce a new major version and restrict Airflow 3 to only use the new ES major version). Since only ElasticsearchTaskHandler leverages the LogTemplate db model, we add the LogTemplate db model in Airflow Core just to support ES and OS providers, but all the other remote log handlers don’t require the LogTemplate db model for either the log read or write path. It seems we couple the ES provider with Airflow Core. For 2. Full compatibility for the ES provider, but much more work needs to be done to support ES under the Airflow 3 security model. Here is the full discussion [4] and more context on how we can solve it properly without a workaround by bypassing the DB session restriction. [1] https://github.com/apache/airflow/pull/53821 [2] https://github.com/apache/airflow/blob/0b968e997c9320bf54e53cdec6fd6c617bbdede4/airflow-core/src/airflow/models/tasklog.py#L27-L28 [3] https://github.com/Owen-CH-Leung/airflow/blob/fix-write-to-es-feature/providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py#L147 [4] https://github.com/apache/airflow/pull/53821#discussion_r2314359557 Looking forward to your feedback! Thanks! Best, Jason
