Yeah, generally when I add operators to the Amazon provider package, my intent is to preserve the behavior of the API,. I'm not claiming that's necessarily the right answer, but that's what i do and I like that idea.
I wouldn't be against the idea of a flag, but I think as a general rule we should assume the default behavior should be familiar to someone who is used to using the API before moving to Airflow without surprises. Of course, his may be an XKCD1172 issue [1] and maybe we SHOULD insist on a standard expectation across providers.... [1] https://xkcd.com/1172/ - ferruzzi ________________________________ From: Shahar Epstein <[email protected]> Sent: Monday, January 12, 2026 12:51 PM To: [email protected] Subject: [EXT] [DISCUSS] Idempotency of "Delete" operators CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. Hey all, For the sake of this discussion, the idempotency of Delete operators means that when an operator is applied to a non-existing resource, it catches a “not found” exception and returns success. By default, the operators I refer to return an exception if the resource is not found. Following this PR #60083 <https://github.com/apache/airflow/pull/60083>, I've realized that we don't handle idempotency consistently in different "Delete" operators across different providers. For example, in some GCP operators there's a tendency to catch "NotFound" exception without any flag (DeletePipelineJobOperator <https://github.com/apache/airflow/blob/bd133c0ebb1219b72f4cb7998b2f5f55c8aff200/providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/pipeline_job.py#L522>), while in some Microsoft's operators we handle it by flag (like "ignore_if_missing" in WASBDeleteBlob <https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/_api/airflow/providers/microsoft/azure/operators/wasb_delete_blob/index.html#module-contents> ). My questions for discussion are as follows: 1. Do we want to require that idempotency for operators (that normally return an error) will be handled by a flag? 2. If we agree on having a flag, what should be the default value of this flag? 3. Given the answer to the last question, should we break the interface of existing operators where applicable? My answers: 1. If the original API usually returns an error, catching NotFound error and returning success "defies" the intent of the original API, and I think that it might be confusing and even limiting in some cases. Therefore, if originally a "Delete" operator fails and we want to catch this exception - I think that we should at least define a flag for that (preferably with a consistent name across all operators). 2. Default behavior should reflect the original intent of the API. 3. Slowly, but carefully - yes: changing both the default value, and the name of the flag to be consistent. I'll be happy to hear more opinions! Shahar
