Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
amoghrajesh commented on issue #51549: URL: https://github.com/apache/airflow/issues/51549#issuecomment-3912377676 This was handled by: https://github.com/apache/airflow/pull/58669 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
amoghrajesh closed issue #51549: Investigate `Timetable` usage for client-server separation URL: https://github.com/apache/airflow/issues/51549 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
amoghrajesh commented on issue #51549: URL: https://github.com/apache/airflow/issues/51549#issuecomment-3889149976 @uranusjr is there anything pending on this one? I think we have handled it already? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
amoghrajesh commented on issue #51549: URL: https://github.com/apache/airflow/issues/51549#issuecomment-3615737792 @uranusjr have we merged all the changes w.r.t this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
uranusjr commented on issue #51549: URL: https://github.com/apache/airflow/issues/51549#issuecomment-3226822679 Most of the timetable logic need to be in the scheduler. So we should kind of do the opposite of SerializedBaseOperator. 1. Existing tiemtable classes in core are scheduler-only. 2. Create timetable “stub” classes that only takes arguments without any scheduler logic. 3. The stubs are serialised, and deserialised in the scheduler into their “real” timetable class counterparts. Custom subclasses are more problematic. I guess for now (Airflow 3.1) we can just make that one single subclass fulfill both roles as a stub and real timetable. Maybe at some point we need to have an Airflow Extension SDK for this, and other things plugin-wise that you can register into the scheduler. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
kaxil commented on issue #51549: URL: https://github.com/apache/airflow/issues/51549#issuecomment-3196269557 Yes ideally. @uranusjr ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
phanikumv commented on issue #51549: URL: https://github.com/apache/airflow/issues/51549#issuecomment-3195526758 Are we still targeting this for 3.1.0? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
kaxil commented on issue #51549: URL: https://github.com/apache/airflow/issues/51549#issuecomment-2999361543 So that has to be on Server side? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
uranusjr commented on issue #51549: URL: https://github.com/apache/airflow/issues/51549#issuecomment-2998754788 Yes it does. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
kaxil commented on issue #51549: URL: https://github.com/apache/airflow/issues/51549#issuecomment-2996272488 If I use a Custom Timetable, does that code currently run in the Scheduler? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
uranusjr commented on issue #51549: URL: https://github.com/apache/airflow/issues/51549#issuecomment-2979233282 Oh yes, it does, for the first ever run. I forgot about that. This can be changed if we need to though. The scheduler is responsible for calculating all later runs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
kaxil commented on issue #51549: URL: https://github.com/apache/airflow/issues/51549#issuecomment-2969784296 > Timetable is kind of the reverse of e.g. dag and asset. It allows customisation, but all the logis is only needed in the scheduler, not the worker nor the dag processor. The main implementation should therefore likely belong in core. However, non-core timetables (e.g. `EventsTimetable`) may belong in the standard provider instead. Doesn't the dag processor use timetable to get the next run time though? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Investigate `Timetable` usage for client-server separation [airflow]
uranusjr commented on issue #51549: URL: https://github.com/apache/airflow/issues/51549#issuecomment-2969281200 Timetable is kind of the reverse of e.g. dag and asset. It allows customisation, but all the logis is only needed in the scheduler, not the worker nor the dag processor. The main implementation should therefore likely belong in core. However, non-core timetables (e.g. `EventsTimetable`) may belong in the standard provider instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] Investigate `Timetable` usage for client-server separation [airflow]
kaxil opened a new issue, #51549: URL: https://github.com/apache/airflow/issues/51549 Analyze current timetable usage in scheduler vs DAG processor. Decide if timetables belong in core, task-sdk, or separate package (`airflow-commons`/`airflow-protocols`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
