bujjibabukatta opened a new pull request, #72879:
URL: https://github.com/apache/airflow/pull/72879
Summary
DataflowJobStatusSensor (and its trigger, DataflowJobStatusTrigger)
previously
required a job_id, forcing users who don't have the job ID readily available
(e.g. jobs launched by external tools that don't push it to XCom) to write
custom operators just to look it up. This PR adds an alternative job_name
parameter so the sensor/trigger can locate and monitor a job by name instead.
Root cause
Dataflow job names are not globally unique, so simply exposing job_name as a
drop-in replacement for job_id isn't safe — multiple jobs (across retries/
backfills) can share a name. The sensor and trigger had no mechanism to
disambiguate between same-named jobs or to handle the case where the named
job hasn't been created yet.
Fix
- DataflowJobStatusSensor and DataflowJobStatusTrigger now accept job_id or
job_name (exactly one required, enforced via exactly_one(), raising
AirflowException otherwise).
- Added DataflowHook.fetch_job_by_name() (sync) and
AsyncDataflowHook.get_job_by_name() (async), which list jobs in the given
project/location, filter to exact name matches, and return the one with
the latest createTime/create_time.
- If no job matching job_name exists yet, the sensor's poke() returns False
and the trigger's run() sleeps and retries, so both keep sensing until the
job appears or the task times out.
- Once resolved, job_id is pinned internally so subsequent logging/polling
is unambiguous.
- job_name added to template_fields so it can be templated
(e.g. job_name="daily-etl-{{ ds_nodash }}").
closes: #72875
Was generative AI tooling used ?
- [X] Yes - Claude
Generated-by: Claude following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
