potiuk commented on code in PR #30255:
URL: https://github.com/apache/airflow/pull/30255#discussion_r1148351671
##########
airflow/task/task_runner/base_task_runner.py:
##########
@@ -46,18 +49,32 @@ class BaseTaskRunner(LoggingMixin):
Invoke the `airflow tasks run` command with raw mode enabled in a
subprocess.
- :param local_task_job: The local task job associated with running the
- associated task instance.
+ :param base_job: The job associated with running the associated task
instance. The job_runner for it
+ should be LocalTaskJobRunner
+ :param job_runner: The job runner associated with running the associated
task instance. This parameter
+ is only used in case base_job is BaseJobPydantic instance - i.e. in
case of db-less mode of
+ Airflow workers. This means that in case you have your own custom
task runner, you should
+ adopt it to handle both BaseJob (with runner being part of the job)
and
+ BaseJobPydantic (with runner passed as a separate parameter). For
non-db-less-mode, the
+ runner is always part of the job, so custom task runners should be
backwards-compatible.
"""
- def __init__(self, local_task_job):
+ def __init__(self, base_job: BaseJob | BaseJobPydantic, job_runner:
LocalTaskJobRunner | None = None):
+ if hasattr(base_job, "job_runner") and job_runner is None:
Review Comment:
I improved it quite a bit. Now `job_runner` is simply a property in
BaseJobPydantic, which means it is not serialized when sent over the wire,
however when needed, it will retrieve it from locally initialized runner (for
LocalTask, Triggerer, DagFileProcessor). This has some (to be fixed later when
we implement AIP-44 for LocalTaskJob methos) some consequences when running
heartbeat_callback (because heartbeat_callback will be run most likely in the
Internal API server, rather than in the task - but this will be a follow-up
work as one of next steps, I do not want to add more in the scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
