Yeah, that’s how I understood it to mean, and then the callback would go to a 
worker (or a trigger) to run the user code.

-a

> On 22 May 2025, at 16:06, Jarek Potiuk <ja...@potiuk.com> wrote:
> 
>> Option 1 as originally proposed in this thread only does the "is a
> callback
> required" check in scheduler -- not running the callback in scheduler
> 
> Ah - then OK. I thought it's also callback execution. Just checking is fine
> in scheduler.
> 
> On Thu, May 22, 2025 at 4:55 PM Daniel Standish
> <daniel.stand...@astronomer.io.invalid> wrote:
> 
>> Option 1 as originally proposed in this thread only does the "is a callback
>> required" check in scheduler -- not running the callback in scheduler.
>> 
>> On Thu, May 22, 2025 at 7:22 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>> 
>>>> So I very strongly vote for Option 1, and if needed make the scheduler
>>> itself more resilient. The Airflow Scheduler _IS_ airflow. Let’s do what
>> we
>>> need to in order to make it more stable, rather than working around a
>>> problem of our own making, whilst also making it operationally more
>> complex
>>> to run.
>>> 
>>> Hey Ash - I forgot to add. Option 1 is against our new security model.
>> This
>>> is essentially DAG author code executed in the scheduler. Ash - do you
>>> think it is possible to avoid that ? For DAG parsing it resulted with
>>> mandatory dag-processor command separated from scheduler, so I am not
>> sure
>>> how we would solve the security issue here? Or maybe there is another
>> idea
>>> on how to solve it? That would be possible if we had deadline callbacks
>>> defined in the plugins, but again - I think the idea was to be able to
>>> provide callbacks by DAG authors (which IMHO is synonymous with "we do
>> not
>>> run it in scheduler".
>>> 
>>> We could potentially run the callbacks in the Dag processor (which we
>>> already did BTW). but I am not sure if this is what we want.
>>> 
>>> J.
>>> 
>>> 
>>> On Thu, May 22, 2025 at 3:40 PM Elad Kalif <elad...@apache.org> wrote:
>>> 
>>>> My comment on the name is for the suggested component that runs the
>>>> workload. It's not about the feature itself. I just suggest a more
>>> generic
>>>> name so if the need comes it would be easier to execute different kind
>> of
>>>> workloads on it (like callbacks).
>>>> 
>>>> As for reuse the Triggerer I am not a fan of that. It serve a
>> completely
>>>> different porpuse and combining both cases may result in poor usage of
>>> auto
>>>> scaling. I don't think alerts/callbacks/other "misc" should compete on
>>> the
>>>> same resources as actual tasks.
>>>> 
>>>> בתאריך יום ה׳, 22 במאי 2025, 16:19, מאת Jarek Potiuk ‏<
>> ja...@potiuk.com
>>>> :
>>>> 
>>>>> How about Option 3) making it part of triggerer.
>>>>> 
>>>>> I think that goes in the direction we've been discussing in the past
>>>> where
>>>>> we have 'generic workload" that we can submit from any of the other
>>>>> components that will be executed in triggerer.
>>>>> 
>>>>> * that would not add too much complexity - no extra process to manage
>>>>> * triggerer is obligatory part of installation now anyway
>>>>> * usually machines today have more processors and triggerer, with its
>>>> event
>>>>> loop does not seem to be too busy in terms of multi-processor usage
>>>> (there
>>>>> are extra processes accessing the DB but still not much I think). It
>>>> could
>>>>> fork another process to run just deadline checks.
>>>>> * re - multi-team it's even easier, triggerer is already going to be
>>>>> "per-team".
>>>>> * we could even rename triggerer to "generic workload processor"
>> (well
>>>>> shorter name, but to indicate that it could process any kind of
>>>> workloads -
>>>>> not only deferred triggers).
>>>>> 
>>>>> Re: comments from Elad:
>>>>> 
>>>>> 1) Naming wise: I think we settled on the name already (looong
>>>> discussion,
>>>>> naming is hard) and I think the scope of it is just really
>> "deadlines"
>>>> (we
>>>>> also wanted to distinguish it from SLA) - i like the name for this
>>>>> particular callback type, but yes - I agree it should be more
>> generic,
>>>> open
>>>>> for any future types of callbacks. If we go for triggerer handling
>>>> "generic
>>>>> workload" - that is IMHO "generic enough" to handle any future
>>> workloads
>>>>> 
>>>>> 2) I believe this is something that could be handled by the callback.
>>>>> Callback could have the option to be able to submit "cancel" request
>>> for
>>>>> the task it is called back for (via task.sdk API)  - but that should
>> be
>>>> up
>>>>> to the one who writes the callback.
>>>>> 
>>>>> J.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Thu, May 22, 2025 at 10:03 AM Elad Kalif <elad...@apache.org>
>>> wrote:
>>>>> 
>>>>>> I prefer option 2 but I have questions.
>>>>>> 1. Naming wise maybe we should prefer a more generic name as I am
>> not
>>>>> sure
>>>>>> if it should be limited to deadlines? (maybe should be shared with
>>>>>> executing callbacks?)
>>>>>> 2. How do you plan to manage the queue of alerts? What happens if
>> the
>>>>>> process is unhealthy while workers continue to execute tasks?
>>>>>> 
>>>>>> On Thu, May 22, 2025 at 12:56 AM Ryan Hatter
>>>>>> <ryan.hat...@astronomer.io.invalid> wrote:
>>>>>> 
>>>>>>> +1 for option 2, primarily because of:
>>>>>>> 
>>>>>>> It would be more robust and resilient, and therefore be able to
>>> run
>>>>> the
>>>>>>>> callbacks *even in presence of certain kinds of issues like the
>>>>>> scheduler
>>>>>>>> being bogged-down*
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, May 21, 2025 at 5:09 PM Kataria, Ramit
>>>>>> <ramit...@amazon.com.invalid
>>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> I’m working with Dennis on Deadline Alerts (AIP-86). I'd like
>> to
>>>>>> discuss
>>>>>>>> implementation approaches for executing callbacks when Deadline
>>>>> Alerts
>>>>>>> are
>>>>>>>> triggered. As you may know, the old SLA feature has been
>> removed,
>>>> and
>>>>>>> we're
>>>>>>>> planning to introduce Deadline Alerts as a replacement in 3.1.
>>>> When a
>>>>>>>> deadline is missed, we need a mechanism to execute callbacks
>>> (which
>>>>>> could
>>>>>>>> be notifications or other actions).
>>>>>>>> 
>>>>>>>> I’ve identified two main approaches:
>>>>>>>> 
>>>>>>>> Option 1: Scheduler-based
>>>>>>>> In this approach, the scheduler would check on a regular
>> interval
>>>> to
>>>>>> see
>>>>>>>> if the earliest deadline has passed and then queue the callback
>>> to
>>>>> run
>>>>>> in
>>>>>>>> an executor (local or remote). The executor would be specified
>>> when
>>>>>>>> creating the deadline alert and if there’s none specified, then
>>> the
>>>>>>> default
>>>>>>>> executor would be used.
>>>>>>>> 
>>>>>>>> Option 2: New DeadlineProcessor process
>>>>>>>> In this approach, there would be a new process similar to
>>>>>>>> triggerer/dag-processor completely independent from the
>> scheduler
>>>> to
>>>>>>> check
>>>>>>>> for deadlines on a regular interval and also run the callbacks
>>>>> without
>>>>>>>> queueing it in another executor.
>>>>>>>> 
>>>>>>>> Multi-team considerations: For multi-team later this year,
>>> option 2
>>>>>> would
>>>>>>>> be relatively simple to implement. However, for option 1, the
>>>>> callbacks
>>>>>>>> would have to run on a remote executor since there would be no
>>>> local
>>>>>>>> executor.
>>>>>>>> 
>>>>>>>> I recommend going with option 2 because:
>>>>>>>> 
>>>>>>>>  *   It would be more robust and resilient, and therefore be
>>> able
>>>> to
>>>>>> run
>>>>>>>> the callbacks even in presence of certain kinds of issues like
>>> the
>>>>>>>> scheduler being bogged-down
>>>>>>>>  *   It would also run the callbacks almost instantly instead
>> of
>>>>>> having
>>>>>>>> to wait for an executor (especially if there’s a long queue of
>>>> tasks
>>>>>> or a
>>>>>>>> cold-start delay)
>>>>>>>>     *   This could be mitigated by implementing a priority
>>> system
>>>>>> where
>>>>>>>> the deadline callbacks are prioritized over regular tasks but
>>> this
>>>>> is a
>>>>>>>> non-trivial problem with my current understanding of Airflow’s
>>>>>>> architecture
>>>>>>>>  *   It would avoid a potential slight increase in workload
>> for
>>>> the
>>>>>>>> scheduler
>>>>>>>>     *   The additional workload in the scheduler for option 1
>>>> would
>>>>> be
>>>>>>>> checking to see if the earliest deadline has passed on a
>> regular
>>>>>> interval
>>>>>>>> 
>>>>>>>> However, it would introduce another process for admins to
>> deploy
>>>> and
>>>>>>>> manage, and also likely require more effort to implement,
>>> therefore
>>>>>>> taking
>>>>>>>> longer to complete.
>>>>>>>> 
>>>>>>>> So, I’d like to hear your thoughts on these approaches,
>> anything
>>> I
>>>>> may
>>>>>>>> have missed and if you agree/disagree with this direction.
>> Thank
>>>> you
>>>>>> for
>>>>>>>> your input!
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> 
>>>>>>>> Ramit Kataria
>>>>>>>> SDE at AWS
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Reply via email to