KafkaOtto opened a new issue, #72504:
URL: https://github.com/apache/airflow/issues/72504
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
`KubernetesPodOperator.init_container_logs` only has an effect when
`deferrable=False`.
In the sync execution path, `execute_sync` calls `await_pod_start` and then
`await_init_containers_completion` (operators/pod.py), which — when
`init_container_logs` is set — calls
`pod_manager.fetch_requested_init_container_logs(...)` to stream the init
containers' logs.
In the deferrable path, `execute_async`/`invoke_defer_method` constructs a
`KubernetesPodTrigger` and passes it `get_logs`, `base_container_name`,
`poll_interval`, etc., but never passes `init_container_logs` or
`init_containers` — the trigger has no such parameter at all. `triggers/pod.py`
has no references to init containers anywhere; it only fetches logs for
`base_container_name` (the main container) via
`fetch_container_logs_before_current_sec`. `trigger_reentry`, which runs after
the task resumes from deferral, also never calls
`await_init_containers_completion`.
As a result, setting `init_container_logs` on the operator has no effect
once `deferrable=True` is set — the parameter is silently accepted but ignored,
and no init container logs appear in the task logs at all.
### What you think should happen instead?
Either:
- When `init_container_logs` is set to True, `KubernetesPodTrigger` should
be extended to fetch/stream logs for all the init containers (mirroring the
sync-path behavior in `await_init_containers_completion` /
`fetch_requested_init_container_logs`), or
- We add a new flag `init_container_logs_when_deferrable`, and let
`KubernetesPodTrigger` allow fetch/stream logs for all the init containers
### Operating System
N/A
### Deployment
None
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
latest
### Official Helm Chart version
1.22.0 (latest released)
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
Related but not duplicate:
- #42498 fixed init container log support for the **sync** path only (fix
landed in #43853); it does not mention or address the deferrable path.
- #56895 is a broader, unrelated feature request about efficiency of log
streaming in deferred mode in general.
Relevant code:
-
`providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py`
(`execute_sync`, `await_init_containers_completion`, `invoke_defer_method`)
-
`providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py`
(`KubernetesPodTrigger.__init__`, `serialize`, `run`)
-
`providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/pod_manager.py`
(`fetch_requested_init_container_logs`, `get_init_container_names`)
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
