I love the idea. If we were to store it in the DB, would we keep a history, or only the latest stats from the most recent dag parsing loop? DAG parsing by default is every 30s right?
On Fri, Jun 14, 2024 at 6:53 AM Jarek Potiuk <[email protected]> wrote: > > I think we still need to enable the ability for DAGs at parse time to > access Variables. > > It's actually DISCOURAGED ;) to access Variables at parse time - though we > have an experimental feature to make it efficient and we discussed whether > to treat it as "good practice" and there was strong opposition to that .... > So I am not actually sure what will be our position on it. > > On Fri, Jun 14, 2024 at 12:50 PM Ash Berlin-Taylor <[email protected]> wrote: > > > > > > > > On 14 Jun 2024, at 10:22, Jarek Potiuk <[email protected]> wrote: > > > > > > think in the future of Airflow 3 where > > > we will have task isolation, having `0` for all the DAGs will be a > > > prerequisite for switching to "task isolation" mode and this could be > > > actually verified in a migration tool. > > > > I think we still need to enable the ability for DAGs at parse time to > > access Variables. > > > > Or at least I am not proposing we remove that ability. (I wouldn’t be > > against it, but I was planning on continuing to support that for now) > > > > -ash >
