per-andersen opened a new issue, #74469:
URL: https://github.com/apache/airflow/issues/74469
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.3.2, main
### What happened and how to reproduce it?
Running a DAG with `DAG.test()` fails for any task whose `task_id` contains a
non ASCII character, as soon as that task pushes a return value XCom. The
task
exhausts its retries and the DAG run ends up failed.
In our case the affected task ids are Danish: `Købsmoms`,
`ResultatOpgørelse`,
`ResultatOpgørelseSammenligning`.
To reproduce, save the following in your DAGs folder and run:
```python
from airflow.sdk import DAG, task
with DAG(dag_id="non_ascii_task_id", schedule=None) as dag:
@task(task_id="Købsmoms", retries=0)
def koebsmoms() -> str:
# Any non-None return triggers the return-value XCom push.
return "value"
koebsmoms()
if __name__ == "__main__":
dag.test()
```
Expected: the run succeeds.
Actual: `UnicodeDecodeError` and the DAG run fails. Without `retries=0`, the
task
retries on every attempt first.
The same DAG with an ASCII `task_id` (e.g. `Kobsmoms`) succeeds.
### What you think should happen instead?
Here is a traceback:
```
Traceback (most recent call last):
File ".../airflow/sdk/execution_time/task_runner.py", line 1598, in run
_push_xcom_if_needed(result, ti, log)
File ".../airflow/sdk/execution_time/task_runner.py", line 2188, in
_push_xcom_if_needed
_xcom_push(ti, BaseXCom.XCOM_RETURN_KEY, result,
mapped_length=mapped_length)
File ".../airflow/sdk/execution_time/task_runner.py", line 883, in
_xcom_push
XCom.set(
File ".../airflow/sdk/bases/xcom.py", line 87, in set
SUPERVISOR_COMMS.send(
File ".../airflow/sdk/execution_time/supervisor.py", line 1949, in send
self.supervisor._handle_request(msg, log, 0)
File ".../airflow/sdk/execution_time/supervisor.py", line 1726, in
_handle_request
resp, dump_opts = handle_set_xcom(self.client, msg)
File ".../airflow/sdk/execution_time/request_handlers.py", line 182, in
handle_set_xcom
client.xcoms.set(
File ".../airflow/sdk/api/client.py", line 634, in set
self.client.post(f"xcoms/{dag_id}/{run_id}/{task_id}/{key}",
params=params, json=value)
...
File ".../httpx/_transports/wsgi.py", line 136, in handle_request
stream = WSGIByteStream(result)
File ".../a2wsgi/asgi.py", line 207, in start_asgi_app
self.app(build_scope(environ), self.asgi_receive, self.asgi_send),
File ".../a2wsgi/asgi.py", line 93, in build_scope
path = root_path + environ.get("PATH_INFO",
"").encode("latin1").decode("utf8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 62:
invalid start byte
```
Byte `0xf8` is `ø`. In our run, position 62 lands exactly on the task_id
inside
`xcoms/{dag_id}/{run_id}/{task_id}/{key}`; the exact position depends on the
length of the dag_id and run_id.
The task instance then goes to `up_for_retry`, repeats on every attempt, and
the
DAG run finishes as `failed`.
Task ids are validated by `validate_key` in
`airflow/sdk/definitions/_internal/node.py`, which uses `KEY_REGEX =
r"^[\w.-]+$"`.
In Python, `\w` matches Unicode letters, so `Købsmoms` is accepted as a valid
`task_id` when the DAG is defined. A key that passes validation should not
then
fail at runtime. Either:
- `DAG.test()` should handle non-ASCII `task_id` values, so the in-process
test
harness is not the one place they break, or
- validation should reject non-ASCII ids at definition time, with a message
that
points to `task_display_name` / `dag_display_name`.
### Operating System
Ubuntu Server 24.04.5 LTS
### Deployment
Other Docker-based deployment
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
Not provider-specific. From the 3.3.2 / Python 3.11 constraints file:
apache-airflow-providers-standard==1.19.0
### Official Helm Chart version
_No response_
### Kubernetes Version
Not Applicable
### Helm Chart configuration
_No response_
### Docker Image customizations
Not Applicable
### Anything else?
Every time a non ASCII character is used in `task_id` of `DAG.test()`.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
