Thanks for your feedback! Someone from the Cloud Composer team will work on it ~this week. We will create an issue on https://github.com/apache/airflow/issues for this change.
- Eugene On Fri, Jun 14, 2024 at 11:47 PM Daniel Standish <[email protected]> wrote: > I wonder if we could generalize to summarize what network activity there > was on dag parse more generally. e.g. if there were API calls or whatnot. > > Quick google search yields this > https://thepythoncode.com/article/make-a-network-usage-monitor-in-python > but not sure it could be used for this. Intuitively it seems possible. > > On Fri, Jun 14, 2024 at 1:45 PM Jarek Potiuk <[email protected]> wrote: > > > Agree with Kaxil. Last run is.good. Actually if we are looking for > Airflow > > 3, the target for 'readu for migration' will be just '0 everywhere' . > > Having last run and getting totals and seeing it goes to 0 steadily > should > > be the 'ready for migration' target for anyone running Airflow 3. > > > > pt., 14 cze 2024, 22:36 użytkownik Kaxil Naik <[email protected]> > > napisał: > > > > > Yeah, it is 30s by default; storing it in DB would be useless, apart > from > > > the most recent run, in my opinion. > > > > > > On Fri, 14 Jun 2024 at 15:24, Constance Martineau > > > <[email protected]> wrote: > > > > > > > I love the idea. If we were to store it in the DB, would we keep a > > > history, > > > > or only the latest stats from the most recent dag parsing loop? DAG > > > parsing > > > > by default is every 30s right? > > > > > > > > On Fri, Jun 14, 2024 at 6:53 AM Jarek Potiuk <[email protected]> > wrote: > > > > > > > > > > I think we still need to enable the ability for DAGs at parse > time > > to > > > > > access Variables. > > > > > > > > > > It's actually DISCOURAGED ;) to access Variables at parse time - > > though > > > > we > > > > > have an experimental feature to make it efficient and we discussed > > > > whether > > > > > to treat it as "good practice" and there was strong opposition to > > that > > > > .... > > > > > So I am not actually sure what will be our position on it. > > > > > > > > > > On Fri, Jun 14, 2024 at 12:50 PM Ash Berlin-Taylor <[email protected] > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > On 14 Jun 2024, at 10:22, Jarek Potiuk <[email protected]> > wrote: > > > > > > > > > > > > > > think in the future of Airflow 3 where > > > > > > > we will have task isolation, having `0` for all the DAGs will > be > > a > > > > > > > prerequisite for switching to "task isolation" mode and this > > could > > > be > > > > > > > actually verified in a migration tool. > > > > > > > > > > > > I think we still need to enable the ability for DAGs at parse > time > > to > > > > > > access Variables. > > > > > > > > > > > > Or at least I am not proposing we remove that ability. (I > wouldn’t > > be > > > > > > against it, but I was planning on continuing to support that for > > now) > > > > > > > > > > > > -ash > > > > > > > > > > > > > > > -- Eugene
