I have some working code saved in my git stash on what Rob suggested. I implemented pre_state_changed_callback and post_state_changed_callback to handle and react to any state change of a dag_run. However, given that dga_run can be updated by the webserver as well, which means executing Python code in the webserver, something which we don't want to do.
On Wed, Jun 16, 2021 at 12:04 AM Canapathy, Subash <[email protected]> wrote: > My 2 cents. Roughly based on conversations with Airflow users internal and > external. > > > > - Platformization and multi-tenancy has been hard to solve on Airflow. > Large deployments of Airflow (multi-cluster on-prem or cloud) have a need > to platformize, manage/govern and vend Airflow as a service to their data > engineering communities internally. The feature/means to achieving this > will be through 1/ DAG manifests and making DAGs a top level > tracked/guid-ed entity that has permissions and context models associated > with it so that multiple tenants can operate out of the same Airflow > environment. > - Expanding DAG folders into workspaces/profiles. Users will benefit > from a high level construct to group DAGs. This will unlock more > opportunities on permission scoping and templatization of access across all > dags in that profile/folder. This might also have some UX benefits as a > side effect – users dislike seeing a thousand DAGs on the list view. > - Modernizing the DAGProcessing->Executor loop in order to support > remote DAG fetcher, synced DAGs and similar features. This will lessen the > reliance to on-disk files as source of truth. While DAG processing can > become faster purely based on DAG Manifest + serialized version of DAG. The > secondary process can asynchronously work to update the manifest and > serialized DAG based on either filesystem (default) or any configured > remote DAG fetcher. > > > > *From: *Ash Berlin-Taylor <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Tuesday, June 15, 2021 at 2:55 AM > *To: *"[email protected]" <[email protected]> > *Subject: *[EXTERNAL] Roadmap ideas for Airflow 2.2 and beyond > > > > *CAUTION*: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > Hi everyone, > > > > As I'm sure many of you are aware I (along with Aizhamal) am giving the > opening keynote at this year's Airflow summit, and I'm covering "what's > next after 2.0" -- essentially what is the roadmap for Airflow for the next > 12-18 months. > > > > Since Airflow is a community project first and foremost I'd like to get > all your ideas, no matter how off the wall :) > > > > I've got my own ideaas, and 2.2 is fairly firm already (AIPs 39 and 40), > but 2.3 and beyond starts to get less clear, so if you have something that > you'd like to see Airflow be able to do or do better, now is the time to > speak up. > > > > You don't have to have a solution, just "I find doing X > hard/annoying/difficult" is enough. > > > > (And a general reminder: the roadmap is a statement of intent, not a > promise of timeline or even that a feature will actually be implemented) > > > > To keep this thread manageable, please can we avoid discussions _*in this > thread*_ about ideas and keep +1/me too's to a minimum. > > > > Cheers, > > -ash >
