xchwan commented on issue #74025: URL: https://github.com/apache/airflow/issues/74025#issuecomment-6598584220
I traced the task-start path before anyone writes more code here. Short
version:
- The task process always needs one full dict (`process_params` merges the
whole conf, templates and user code expect a dict). So #74392 / #75935 can only
save the API server and supervisor copies, not the task one.
- Changing what the task sees by default would break existing Dags
(`isinstance(conf, dict)`, `json.dumps(conf)`, `{{ dag_run.conf }}`, params
overrides).
- Passing conf around as an opaque string only works while no layer reads
it. Non-Python SDKs and older Execution API clients need a dict, so the saving
disappears for them, and any future feature that wants to look at conf in the
server or supervisor hits the same wall.
- #74048 showed the task side paying about what the supervisor saved, so the
net win is unclear.
One idea that doesn't break anything: an opt-in `conf_keys` on the Dag. The
server only loads those keys (plus the ones named in params) in SQL, and any
other key is fetched on demand, so a missing declaration is slower, not broken.
Dags without it behave as today. It needs serialization, a subset query per
database, and an on-demand fetch endpoint.
Is that worth a PR, or would you rather just document "keep conf small, pass
references" and leave the startup path alone?
---
Drafted-by: Claude Code (Sonnet 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
