dabla opened a new pull request, #74675: URL: https://github.com/apache/airflow/pull/74675
A retry of `PowerBIDatasetRefreshOperator`, for instance after a worker crash, starts a new dataset refresh while the first one may still be running. A retry of `MSGraphAsyncOperator` sends its request again and fetches every page from the first one. There is a second gap: the refresh is started by the trigger, and its id lives only in the trigger's memory until the first event. So a triggerer restart in that window starts a second refresh as well. This builds on `ResumableJobMixin` from `common.compat` (#74673). **PowerBI refresh** - The refresh is now started from the worker. - Its id is kept in the task state store before the task defers to poll. - A retry reconnects to a refresh still in progress and does not start a new one once it has completed. - A refresh that failed, or that dropped out of the dataset's refresh history, is started again. - Since the polling trigger gets the id from the start, a triggerer restart no longer starts a second refresh, on any Airflow version. - A failing poll no longer cancels the refresh when the operator is durable, so the retry can pick it up. - As far as I can see, this is the first deferrable operator built on the mixin: `poll_until_complete` defers, and the mixin has already stored the id by then. **MSGraph** - New opt-in `durable` flag. Every processed page and the link to the next one go to the task state store, so a retry continues at the page it stopped at. - A request that is not a `GET` keeps its result, so a retry returns it instead of sending the request again. - It is opt-in because pages are stored as they are, and Graph pages can be larger than the store's default value limit. Resuming needs Airflow 3.3. On older versions, both operators behave as before. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
