vatsrahul1001 opened a new pull request, #68819:
URL: https://github.com/apache/airflow/pull/68819
### Problem
When the scheduler runs **in-cluster**, the kubernetes Python client
attaches an in-cluster `Configuration` to every `V1Pod`, and that
`Configuration.refresh_api_key_hook` is a **local closure**
(`InClusterConfigLoader._set_config.<locals>._refresh_api_key`). `pickle`
cannot serialize a local closure.
`KubernetesExecutor` puts each task's `KubernetesJob` (which embeds the
task's `pod_override` `V1Pod`) onto a `multiprocessing.Manager` queue, whose
`.put()` pickles synchronously in the scheduler. So any in-cluster deployment
where a task sets a `V1Pod` `pod_override` crashes the scheduler in a loop:
```
_pickle.PicklingError: Can't pickle local object
<function InClusterConfigLoader._set_config.<locals>._refresh_api_key at
...>
when serializing kubernetes.client.configuration.Configuration ...
when serializing dict item 'pod_override'
when serializing ... ExecuteTask ...
File ".../executors/kubernetes_executor.py", line 230, in execute_async
self.task_queue.put(KubernetesJob(key, command, kube_executor_config,
pod_template_file))
```
This is **independent of the Airflow version** — it is gated on having a
`V1Pod` `pod_override` and an in-cluster scheduler. It surfaces with kubernetes
client **36.x** (whose `Configuration` carries the unpicklable hook), and
pinning the client is not a viable fix for everyone (35.x has its own
[`no_proxy` bug
#2520](https://github.com/kubernetes-client/python/issues/2520), so deployments
that need 36.x for that fix cannot downgrade).
### Fix
Serialize the `pod_override` to a plain dict (dropping the `Configuration`)
**before** it is queued, using the existing `PodGenerator.serialize_pod`, and
rebuild the `V1Pod` worker-side in `run_next` via
`PodGenerator.deserialize_model_dict`. The worker already reconstructs its own
kube client, so nothing is lost. The worker uses `kube_executor_config` for the
pod override; the same `V1Pod` is also referenced by the workload's
`executor_config`, so that copy is sanitized too (it is otherwise pickled as
part of the workload).
This keeps `pod_override` working regardless of the kubernetes client
version, rather than depending on a client version whose `Configuration`
happens to be picklable. The worker side already sanitizes pods with
`sanitize_for_serialization` (kubernetes_executor_utils.py:492) — this closes
the same gap on the scheduler→queue boundary.
### Tests
- New regression test `test_execute_async_queues_picklable_pod_override`:
builds a `pod_override` whose `Configuration` carries an unpicklable
local-closure hook, asserts the raw pod is unpicklable (reproduces the crash),
then asserts the queued `KubernetesJob` pickles cleanly and the serialized dict
round-trips back to an equivalent `V1Pod`.
- Updated `test_pod_template_file_override_in_executor_config` to reflect
the queued `kube_executor_config` now being a dict; its existing `run_next`
round-trip assertion (the reconstructed pod) is unchanged and still passes.
- Full `test_kubernetes_executor.py` suite passes (111 passed, 1 skipped
AF<3.2-only).
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
