+1 to what Daniel wrote. On Sun, Sep 14, 2025 at 8:11 PM Daniel Standish via dev < [email protected]> wrote:
> I would like it if we drop the log template model entirely. It’s also > possible to write to TI field. And better to not allow any context var in > log template. > > On Sun, Sep 14, 2025 at 12:01 AM Zhe-You Liu <[email protected]> wrote: > > > Hi all, > > > > While reviewing “Fix the write-to-es feature #53821” [1], I found that > the > > ElasticsearchTaskHandler (and OpensearchTaskHandler as well) has been > > broken for a long time since Airflow 3, as we block the DB session, but > > ElasticsearchTaskHandler still depends on the LogTemplate [2] db model > for > > the log writing path [3]. To fix the ElasticsearchTaskHandler in the > > Airflow 3 security model, we can either: > > > > 1. > > > > Drop LogTemplate db model support for ElasticsearchTaskHandler, > > > > or > > > > 2. > > > > Still support the LogTemplate feature with ElasticsearchTaskHandler by > > fetching the LogTemplate db model before we block the DB session and > > attaching the LogTemplate attribute to the ExecuteTask workload. > > > > For 1. > > > > It will be much simpler and will also decouple the ES provider from > Airflow > > Core, in my opinion, but with no backward compatibility (we need to > > introduce a new major version and restrict Airflow 3 to only use the new > ES > > major version). > > > > Since only ElasticsearchTaskHandler leverages the LogTemplate db model, > we > > add the LogTemplate db model in Airflow Core just to support ES and OS > > providers, but all the other remote log handlers don’t require the > > LogTemplate db model for either the log read or write path. It seems we > > couple the ES provider with Airflow Core. > > > > For 2. > > > > Full compatibility for the ES provider, but much more work needs to be > done > > to support ES under the Airflow 3 security model. > > > > Here is the full discussion [4] and more context on how we can solve it > > properly without a workaround by bypassing the DB session restriction. > > > > [1] https://github.com/apache/airflow/pull/53821 > > [2] > > > > > https://github.com/apache/airflow/blob/0b968e997c9320bf54e53cdec6fd6c617bbdede4/airflow-core/src/airflow/models/tasklog.py#L27-L28 > > [3] > > > > > https://github.com/Owen-CH-Leung/airflow/blob/fix-write-to-es-feature/providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py#L147 > > [4] https://github.com/apache/airflow/pull/53821#discussion_r2314359557 > > > > Looking forward to your feedback! Thanks! > > > > Best, > > Jason > > >
