> I don't really think of this as "breaking with the API", or think there is much of a conflict
Quite agree, "Airflow Provider API' does not necessarily need to be 1-1 mapping to the "original" API. The changes we might introduce might cause breaking changes in the "Provider APIs" - and cause major version bumps with those - but that's a different story. J. On Thu, Jan 15, 2026 at 11:00 PM Daniel Standish via dev < [email protected]> wrote: > Maybe file what i'm about to say under superfluous commentary but... > > I don't really think of this as "breaking with the API", or think there is > much of a conflict. The API is just there for you to use. It has its own > behavior. You understand how it works in order to do what you need to do. > But *you* decide what you need to do, not the API. Sometimes that means > you handle errors it throws at you. > > Operators are a different layer of abstraction and governed by user needs > and not design choices of the API it is using. > > So just because the API throws an error in a certain operation, perhaps > even an operation that has a name similar to your operator, doesn't mean > your code should explode when you are trying to use that operation. > > > > On Thu, Jan 15, 2026 at 1:21 PM Shahar Epstein <[email protected]> wrote: > > > As I reviewed a PR that fixed related behavior, I didn't know what the > > appropriate approach is actually. I think that this confusion is > justified > > due to the conflict between achieving idempotency and breaking with the > > original API (unlike other kinds of operators that typically follow it). > > Enforcing might be hard, but I think that it should be documented as part > > of guidelines for contributors and maintainers. Also, if we agree that > this > > is the desired behavior for most cases - deprecating the flags where > > applicable, and aligning the relevant operators sounds like a reasonable > > thing to do. > > > > > > Shahar > > > > > > On Tue, Jan 13, 2026 at 11:15 PM Oliveira, Niko <[email protected]> > > wrote: > > > > > I think I agree with the crowd here, we shouldn't make a fuss if the > > > resource is already deleted. > > > But I would also strongly +1 Jarek's second point. Is this consistency > we > > > really need to do a lot of work to enact/enforce? Are users actually > > > struggling even with this? Or have we so far settled on what makes the > > most > > > sense for each operator and not many folks are perfectly happy with the > > > current state? > > > > > > > > > Cheers, > > > Niko > > > > > > ________________________________ > > > From: Shahar Epstein <[email protected]> > > > Sent: Tuesday, January 13, 2026 1:03:11 PM > > > To: [email protected] > > > Subject: RE: [EXT] [DISCUSS] Idempotency of "Delete" operators > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > > que > > > le contenu ne présente aucun risque. > > > > > > > > > > > > Your wish is Gemini's command - I tried to scan the entire repo for > > Delete > > > operators with such flags. Luckily there aren't many + we are not that > > > creative in naming :) > > > https://github.com/apache/airflow/pull/60483 > > > > > > I've read the statements in the thread until now, and I'm not strongly > > > against making all operators idempotent by default without a flag, as > > > customizations could be made by subclassing + extra "if exist" > statement. > > > It's more important for me that the behavior will be mostly consistent > in > > > such operators (with granted exemptions, where applicable). This would > > make > > > it clearer for contributors and maintainers as it breaks with the API, > > > which is not trivial IMO. > > > > > > I'll give this discussion a couple of more days, and if there are no > > strong > > > objections against the apparent consensus, I'll start a lazy consensus > > > thread + create a GitHub issue to align all relevant operators. > > > > > > > > > Shahar > > > > > > On Tue, Jan 13, 2026 at 9:07 PM Jarek Potiuk <[email protected]> wrote: > > > > > > > I think yes - we should strive to have the delete "kinds" of > operations > > > to > > > > silently ignore already deleted content. We can agree on it as a > > > "default" > > > > behaviour - and probably have it written somewhere in best practices > > for > > > > writing operators. > > > > > > > > But - we also should be pragmatic in the sense that a) there might be > > > > (justified) exceptions b) it's not very likely we will be able to > > > > consistently and automatically enforce it, and it almost guarantees > > that > > > at > > > > any point in time some of those will be following different > semantics - > > > > because we forget about it over time or won't realise this is > > happening. > > > > > > > > I wish we had some way of enforcing or at least detecting it :) > > > > > > > > On Tue, Jan 13, 2026 at 7:17 PM Daniel Standish via dev < > > > > [email protected]> wrote: > > > > > > > > > Yeah just suppress the exception. > > > > > > > > > > Task should not fail cus object already deleted. > > > > > > > > > > The intent is that the thing is not there, not to verify that it > was > > > and > > > > > that you deleted it. > > > > > > > > > > If users want diff behavior they can subclass. > > > > > > > > > > Simpler is better than complex, re the flag idea. Does not seem > like > > a > > > > case > > > > > for a global param. > > > > > > > > > > > > > > > > > > > > On Tue, Jan 13, 2026 at 9:49 AM Ash Berlin-Taylor <[email protected]> > > > > wrote: > > > > > > > > > > > Given then general rule of idempontency of Airflow operators, I > > think > > > > we > > > > > > should break with the API. The intent of a “Delete” operator is > to > > > > ensure > > > > > > the resource no longer exists. So not found is not an error in > that > > > > > model. > > > > > > This makes such operators safe to retry. If it threw an error > when > > it > > > > was > > > > > > deleted then they are not safe to retry, and this is not the > model > > > > > Airflow > > > > > > operators should have. > > > > > > > > > > > > No flag should be needed — if someone doesn’t want this behavoiur > > > than > > > > I > > > > > > would say they should use the hook or API directly. The point of > > the > > > > > > pre-built operators is to package up behaviour like this in a > > > > consistent > > > > > > manner. > > > > > > > > > > > > My 2c: we should standardised on remove the “ignore_if_missing” > > flags > > > > on > > > > > > operators and always catching not exists style errors in > operators > > > and > > > > > > treating those as success. > > > > > > > > > > > > -ash > > > > > > > > > > > > > On 12 Jan 2026, at 22:06, Ferruzzi, Dennis < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > Yeah, generally when I add operators to the Amazon provider > > > package, > > > > my > > > > > > intent is to preserve the behavior of the API,. I'm not claiming > > > > that's > > > > > > necessarily the right answer, but that's what i do and I like > that > > > > idea. > > > > > > > > > > > > > > I wouldn't be against the idea of a flag, but I think as a > > general > > > > rule > > > > > > we should assume the default behavior should be familiar to > someone > > > who > > > > > is > > > > > > used to using the API before moving to Airflow without surprises. > > Of > > > > > > course, his may be an XKCD1172 issue [1] and maybe we SHOULD > insist > > > on > > > > a > > > > > > standard expectation across providers.... > > > > > > > > > > > > > > [1] https://xkcd.com/1172/ > > > > > > > > > > > > > > > > > > > > > - ferruzzi > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > From: Shahar Epstein <[email protected]> > > > > > > > Sent: Monday, January 12, 2026 12:51 PM > > > > > > > To: [email protected] > > > > > > > Subject: [EXT] [DISCUSS] Idempotency of "Delete" operators > > > > > > > > > > > > > > CAUTION: This email originated from outside of the > organization. > > Do > > > > not > > > > > > click links or open attachments unless you can confirm the sender > > and > > > > > know > > > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un > expéditeur > > > > > > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce > jointe > > si > > > > > vous > > > > > > ne pouvez pas confirmer l’identité de l’expéditeur et si vous > > n’êtes > > > > pas > > > > > > certain que le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hey all, > > > > > > > > > > > > > > For the sake of this discussion, the idempotency of Delete > > > operators > > > > > > means > > > > > > > that when an operator is applied to a non-existing resource, it > > > > > catches a > > > > > > > “not found” exception and returns success. By default, the > > > operators > > > > I > > > > > > > refer to return an exception if the resource is not found. > > > > > > > > > > > > > > Following this PR #60083 < > > > > https://github.com/apache/airflow/pull/60083 > > > > > >, > > > > > > > I've realized that we don't handle idempotency consistently in > > > > > different > > > > > > > "Delete" operators across different providers. For example, in > > some > > > > GCP > > > > > > > operators there's a tendency to catch "NotFound" exception > > without > > > > any > > > > > > flag > > > > > > > (DeletePipelineJobOperator > > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/bd133c0ebb1219b72f4cb7998b2f5f55c8aff200/providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/pipeline_job.py#L522 > > > > > > >), > > > > > > > while in some Microsoft's operators we handle it by flag (like > > > > > > > "ignore_if_missing" in WASBDeleteBlob > > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/_api/airflow/providers/microsoft/azure/operators/wasb_delete_blob/index.html#module-contents > > > > > > > > > > > > > > ). > > > > > > > > > > > > > > My questions for discussion are as follows: > > > > > > > 1. Do we want to require that idempotency for operators (that > > > > normally > > > > > > > return an error) will be handled by a flag? > > > > > > > 2. If we agree on having a flag, what should be the default > value > > > of > > > > > this > > > > > > > flag? > > > > > > > 3. Given the answer to the last question, should we break the > > > > interface > > > > > > of > > > > > > > existing operators where applicable? > > > > > > > > > > > > > > My answers: > > > > > > > 1. If the original API usually returns an error, catching > > NotFound > > > > > error > > > > > > > and returning success "defies" the intent of the original API, > > and > > > I > > > > > > think > > > > > > > that it might be confusing and even limiting in some cases. > > > > Therefore, > > > > > if > > > > > > > originally a "Delete" operator fails and we want to catch this > > > > > exception > > > > > > - > > > > > > > I think that we should at least define a flag for that > > (preferably > > > > > with a > > > > > > > consistent name across all operators). > > > > > > > 2. Default behavior should reflect the original intent of the > > API. > > > > > > > 3. Slowly, but carefully - yes: changing both the default > value, > > > and > > > > > the > > > > > > > name of the flag to be consistent. > > > > > > > > > > > > > > I'll be happy to hear more opinions! > > > > > > > > > > > > > > > > > > > > > Shahar > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > >
