atikulmunna commented on issue #70296:
URL: https://github.com/apache/airflow/issues/70296#issuecomment-5128463294
I had a dig through the two `dataproc.py` entries and I don't think either
one fits the "move it into `execute()`" pattern. Writing up what I found before
someone sinks a weekend into them.
Findings on current `main`:
```
DataprocCreateClusterOperator
line 734: kwargs["project_id"] = project_id (project_id)
DataprocSubmitJobOperator
line 2011: "job": self.job, (self.job)
line 2012: "project_id": self.project_id, (self.project_id)
line 2013: "region": self.region, (self.region)
line 2014: "gcp_conn_id": self.gcp_conn_id, (self.gcp_conn_id)
line 2015: "impersonation_chain": self.impersonation_chain,
(self.impersonation_chain)
line 2018: "request_id": self.request_id, (self.request_id)
```
**`DataprocSubmitJobOperator`**
All six reads populate `start_trigger_args.trigger_kwargs` inside `if
self.deferrable and self.start_from_trigger`. The trouble is there's nowhere to
move them to.
`TaskInstance.defer_task()` reads `self.task.start_trigger_args` and drops
`trigger_kwargs` straight into the `Trigger` row. It's called from
`DagRun.schedule_tis()`, so it runs in the scheduler against the serialized
task, and nothing renders on the way. `_validate_start_from_trigger_kwargs()`
only rejects callables. `expand_start_trigger_args(context=...)` looks like it
was meant to be the seam for exactly this, but as far as I can tell it's
defined in four places and never actually called.
So right now, with `start_from_trigger=True`, those fields reach
`DataprocSubmitJobDirectTrigger` as raw Jinja and `execute()` never runs at
all. Moving the block into `execute()` would just quietly disable
`start_from_trigger` instead. This looks like the same underlying gap as #70284
on `DateTimeSensorAsync`, which makes me think it wants fixing in core rather
than operator by operator.
**`DataprocCreateClusterOperator`**
Line 734 hands `project_id` to `ClusterGenerator(**kwargs).make()`, which
interpolates it into URI strings (`zone_uri`, the machine type URIs, the custom
image ones). Those land inside `cluster_config`, which is itself a template
field that Jinja recurses into. So a templated `project_id` does render
correctly today, just not for a reason anyone designed.
Deferring the `ClusterGenerator` call doesn't get you out of it either. The
loose keyword args it needs aren't template fields and aren't in
`get_serialized_fields()`, so stashing them on the instance in `__init__` won't
survive serialization to the worker. The nasty part is that in-process unit
tests would still pass, so it would look fine and then break in a real
deployment. Doing it properly means adding a serialized field to a keyword path
that's already set for removal on 2026-10-05.
**Suggestion**
Could these two be pulled out of the burn-down, or at least flagged in the
issue so they don't read as good first issues? Happy to take either one on if
there's a direction, particularly on whether the `start_from_trigger` rendering
gap should be handled in core.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
