Hey all, For the sake of this discussion, the idempotency of Delete operators means that when an operator is applied to a non-existing resource, it catches a “not found” exception and returns success. By default, the operators I refer to return an exception if the resource is not found.
Following this PR #60083 <https://github.com/apache/airflow/pull/60083>, I've realized that we don't handle idempotency consistently in different "Delete" operators across different providers. For example, in some GCP operators there's a tendency to catch "NotFound" exception without any flag (DeletePipelineJobOperator <https://github.com/apache/airflow/blob/bd133c0ebb1219b72f4cb7998b2f5f55c8aff200/providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/pipeline_job.py#L522>), while in some Microsoft's operators we handle it by flag (like "ignore_if_missing" in WASBDeleteBlob <https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/_api/airflow/providers/microsoft/azure/operators/wasb_delete_blob/index.html#module-contents> ). My questions for discussion are as follows: 1. Do we want to require that idempotency for operators (that normally return an error) will be handled by a flag? 2. If we agree on having a flag, what should be the default value of this flag? 3. Given the answer to the last question, should we break the interface of existing operators where applicable? My answers: 1. If the original API usually returns an error, catching NotFound error and returning success "defies" the intent of the original API, and I think that it might be confusing and even limiting in some cases. Therefore, if originally a "Delete" operator fails and we want to catch this exception - I think that we should at least define a flag for that (preferably with a consistent name across all operators). 2. Default behavior should reflect the original intent of the API. 3. Slowly, but carefully - yes: changing both the default value, and the name of the flag to be consistent. I'll be happy to hear more opinions! Shahar
