will be undocumented*
On Tuesday, September 16th, 2025 at 5:01 PM, asquator <asqua...@proton.me>
wrote:
> I see the motivation, but does it have to look so bad?
>
> The subclass will look like this:
>
> class SchedulerJobRunnerLinearTIScan(SchedulerJobRunner):
> def init(
> self,
> job: Job,
> num_runs: int = conf.getint("scheduler", "num_runs"),
> scheduler_idle_sleep_time: float = conf.getfloat("scheduler",
> "scheduler_idle_sleep_time"),
> log: Logger | None = None,
> ):
> super().init(
> job=job,
> num_runs=num_runs,
> scheduler_idle_sleep_time=scheduler_idle_sleep_time,
> log=log,
> )
> self.task_selector = TASK_SELECTORS[LINEAR_SCAN_SELECTOR]
>
> The super class will use the injected hard-coded task selector.
>
> Can't we introduce a configuration hierarchy like `core.internal` and put
> there things not exposed to the end user? So we don't have to do this weird
> subclassing?
>
> It will look thus:
>
> class SchedulerJobRunner(...):
> task_selector_type = conf.get("scheduler.internal", "task_selector_strategy")
> self.task_selector = TASK_SELECTORS[task_selector_type]
>
> We'd just like to have an internal toggle as an implementation detail, which
> won't be undocumented and custom implementations won't be supported. It's
> just more convenient and straightforward.
>
> Mb there's another way of internal settings management I missed?
>
>
> On Tuesday, September 16th, 2025 at 11:34 AM, Ash Berlin-Taylor
> a...@apache.org wrote:
>
> > > On 16 Sep 2025, at 08:58, asquator asqua...@proton.me wrote:
> > >
> > > Yes, exposing pluggable features means fixing an API, which is confining
> > > and just hard to do given the current implementation
> >
> > class MyScheduler:
> > def execute(self):
> > while True:
> > # Do what ever you want.
> >
> > `airflow scheduler --impl=my.module.MyScheduler`
> >
> > That is the API.
> >
> > That is as pluggable as we need it to be.
> >
> > Everything can be built on top of that, including if you want it, a
> > pluggable task selection mechanisms.
> >
> > Airflow already has too many config options and ways of tuning behaviour.
> > We need less of them, not more.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
