Given then general rule of idempontency of Airflow operators, I think we should break with the API. The intent of a “Delete” operator is to ensure the resource no longer exists. So not found is not an error in that model. This makes such operators safe to retry. If it threw an error when it was deleted then they are not safe to retry, and this is not the model Airflow operators should have.
No flag should be needed — if someone doesn’t want this behavoiur than I would say they should use the hook or API directly. The point of the pre-built operators is to package up behaviour like this in a consistent manner. My 2c: we should standardised on remove the “ignore_if_missing” flags on operators and always catching not exists style errors in operators and treating those as success. -ash > On 12 Jan 2026, at 22:06, Ferruzzi, Dennis <[email protected]> wrote: > > Yeah, generally when I add operators to the Amazon provider package, my > intent is to preserve the behavior of the API,. I'm not claiming that's > necessarily the right answer, but that's what i do and I like that idea. > > I wouldn't be against the idea of a flag, but I think as a general rule we > should assume the default behavior should be familiar to someone who is used > to using the API before moving to Airflow without surprises. Of course, his > may be an XKCD1172 issue [1] and maybe we SHOULD insist on a standard > expectation across providers.... > > [1] https://xkcd.com/1172/ > > > - ferruzzi > > > ________________________________ > From: Shahar Epstein <[email protected]> > Sent: Monday, January 12, 2026 12:51 PM > To: [email protected] > Subject: [EXT] [DISCUSS] Idempotency of "Delete" operators > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne > cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas > confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le > contenu ne présente aucun risque. > > > > Hey all, > > For the sake of this discussion, the idempotency of Delete operators means > that when an operator is applied to a non-existing resource, it catches a > “not found” exception and returns success. By default, the operators I > refer to return an exception if the resource is not found. > > Following this PR #60083 <https://github.com/apache/airflow/pull/60083>, > I've realized that we don't handle idempotency consistently in different > "Delete" operators across different providers. For example, in some GCP > operators there's a tendency to catch "NotFound" exception without any flag > (DeletePipelineJobOperator > <https://github.com/apache/airflow/blob/bd133c0ebb1219b72f4cb7998b2f5f55c8aff200/providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/pipeline_job.py#L522>), > while in some Microsoft's operators we handle it by flag (like > "ignore_if_missing" in WASBDeleteBlob > <https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/_api/airflow/providers/microsoft/azure/operators/wasb_delete_blob/index.html#module-contents> > ). > > My questions for discussion are as follows: > 1. Do we want to require that idempotency for operators (that normally > return an error) will be handled by a flag? > 2. If we agree on having a flag, what should be the default value of this > flag? > 3. Given the answer to the last question, should we break the interface of > existing operators where applicable? > > My answers: > 1. If the original API usually returns an error, catching NotFound error > and returning success "defies" the intent of the original API, and I think > that it might be confusing and even limiting in some cases. Therefore, if > originally a "Delete" operator fails and we want to catch this exception - > I think that we should at least define a flag for that (preferably with a > consistent name across all operators). > 2. Default behavior should reflect the original intent of the API. > 3. Slowly, but carefully - yes: changing both the default value, and the > name of the flag to be consistent. > > I'll be happy to hear more opinions! > > > Shahar --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
