meel-ayush opened a new issue, #76550:
URL: https://github.com/apache/airflow/issues/76550
### Under which category would you file this issue?
Task SDK
### Apache Airflow version
3.4.0rc3 (and main)
### What happened and how to reproduce it?
### Description
When an IPC request frame sent from a task runner fails validation or
deserialization on the supervisor (for example, due to a malformed payload,
schema drift, or unrecognized discriminator), `handle_requests` logs the
exception and runs `continue` without returning a response frame.
Because `CommsDecoder.send()` on the task runner blocks synchronously on
`socket.recv(4)` waiting for a response, the child process deadlocks
permanently. At the same time, the supervisor continues its event loop and
sends regular heartbeats to the API server, so the scheduler's zombie detector
never fires. The task stays stuck in `RUNNING` and holds worker capacity until
manually killed.
### Affected Code
In `task-sdk/src/airflow/sdk/execution_time/supervisor.py:1176-1180`:
```python
try:
msg =
self.decoder.validate_python(self._deserialize_request(request.body))
except Exception:
log.exception("Unable to decode message", body=request.body)
continue
```
And in `task-sdk/src/airflow/sdk/execution_time/comms.py:347-348`:
```python
frame = await asyncio.to_thread(self._read_frame)
return self._from_frame(frame)
```
### Steps to Reproduce
1. Set up an `ActivitySubprocess` over a socketpair.
2. Send a request frame with an invalid body (e.g. `{"type": "GetVariable"}`
without required keys).
3. The supervisor logs `Unable to decode message` and loops.
4. The child process thread hangs on `recv(4)` waiting for
`_ResponseFrame(id=request.id)`.
5. The supervisor continues heartbeating the API server.
Minimal reproduction script:
```python
import socket
import uuid
from unittest.mock import Mock
import msgspec
from airflow.sdk.execution_time.comms import _RequestFrame
from airflow.sdk.execution_time.supervisor import (
ActivitySubprocess,
length_prefixed_frame_reader,
)
read_end, write_end = socket.socketpair()
ti_id = uuid.uuid4()
mock_client = Mock()
subprocess = ActivitySubprocess(
id=ti_id,
socket=read_end,
client=mock_client,
dag_rel_path="test_dag.py",
bundle_info=Mock(),
)
gen = subprocess.handle_requests(log=Mock())
frame_reader = length_prefixed_frame_reader(
gen, on_close=subprocess._on_socket_closed
)
req = _RequestFrame(id=101, body={"type": "InvalidMessage"})
encoder = msgspec.msgpack.Encoder()
payload = encoder.encode(req)
write_end.sendall(len(payload).to_bytes(4, "big") + payload)
frame_reader(read_end)
write_end.settimeout(2.0)
try:
write_end.recv(4)
except TimeoutError:
print(
"DEADLOCK: Child blocked waiting for response frame that was never
sent"
)
```
### Expected Behavior
If the supervisor cannot deserialize or validate `request.body`, it should
send back an error response frame (`_ResponseFrame(id=request.id, error=...)`)
so the task runner receives an error and fails or retries the task, rather than
deadlocking the process.
### Proposed Fix
In `supervisor.py:handle_requests`, send an error response before continuing:
```python
try:
msg =
self.decoder.validate_python(self._deserialize_request(request.body))
except Exception as e:
log.exception("Unable to decode message", body=request.body)
err_resp = _ResponseFrame(
id=request.id,
error={"detail": f"Supervisor decoding failed: {e}"},
)
self._send_response(err_resp)
continue
```
I have reproduced this with the test above and am working on a PR with unit
tests to fix this.
### What you think should happen instead?
_No response_
### Operating System
Kali GNU/Linux
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]