Yup, Amogh already did that 3 weeks ago :) https://github.com/astronomer/agents/pull/222
On Fri, 31 Jul 2026 at 15:56, Jarek Potiuk <[email protected]> wrote: > Suggestion: > > Maybe we should add agentic suggestions / SKILLS to do it and even run some > agentic checks and create an issue pointing to some of those? This might > then kick in when either author or reviewer use agents to do what they are > doing. > I think those are far more effective nowadays than relying on people a) > reading it b) remembering that it's there, or c) implementing it :) > > J. > > On Fri, Jul 31, 2026 at 8:30 AM Amogh Desai <[email protected]> wrote: > > > Hi All, > > > > Through this email, I am trying to direct you to a better pattern to > reduce > > your operational overhead of having to > > maintain some custom piece of code that you own across airflow upgrades. > > > > I came across a few users on r/apache_airflow > > < > > > https://www.reddit.com/r/apache_airflow/comments/1vatr6h/looking_for_feedback_is_anyone_using_task_state/ > > > > > who > > are running custom submit + sensor task groups for > > external jobs such as dbt cloud / databricks / snowflake jobs: one task > > submits, a separate task polls for completion. > > > > It got me curious and I built the same pattern for Databricks and want to > > share why task state store (AIP-103, shipped in 3.3) > > is the better fit. > > > > My setup looked like this: > > > > submit = DatabricksSubmitRunOperator( > > task_id="submit", > > wait_for_termination=False, > > tasks=[...], > > ) > > # custom sensor I wrote > > sense = DatabricksRunSensor( > > task_id="sense", > > run_id=submit.output["run_id"], > > poke_interval=15, > > mode="reschedule", > > ) > > submit >> sense > > > > It works, but it has two real problems. > > > > a) (Semantic) The submit task tells you nothing about whether the job > > succeeded, only that it started, you have to look at sense > > for the actual outcome. > > b) The sensor's own timeout runs on a clock separate from the job itself, > > so it can fail the task even after the job completed successfully > > underneath it. > > > > Task state store collapses this into one task: > > > > submit = DatabricksSubmitRunOperator( > > task_id="submit_and_wait", > > tasks=[...], > > durable=True, > > retries=3, > > ) > > > > durable=True writes the run id to task state store the moment the job > > exists, before the task itself completes. If the worker dies right after > > submission, a retry reads that stored id back and reconnects to the same > > run instead of resubmitting. And because the whole submit-and-wait > > sequence lives in one task, its final status is the job's real outcome. > > Retries are ordinary retries, each one re-checks the actual job status > from > > the > > source of truth rather than tracking a separate poke/timeout state. > > > > Net result: > > - one task instead of two, > > - no custom sensor to maintain, > > - and the crash-safety guarantee. > > > > If you are running a submit+sensor setup like mine for any provider, I > > would like to know whether this covers your case or whether something is > > still missing. (feel free to ping me on Airflow Slack and we can chat) > > > > Thanks & Regards, > > Amogh Desai > > >
