fabbuc-gyg opened a new pull request, #70385: URL: https://github.com/apache/airflow/pull/70385
## What Strip characters that Postgres `jsonb` cannot store (NUL bytes and unpaired UTF-16 surrogates) from a failed Databricks task's error text, in both `extract_failed_task_errors` and `extract_failed_task_errors_async`. ## Why `DatabricksExecutionTrigger` puts each failed task's error (taken from `hook.get_run_output(...)["error"]`, i.e. arbitrary external text) into its `TriggerEvent`. Airflow persists that event payload into the deferred task instance's `next_kwargs`, which is a Postgres `jsonb` column. Databricks run output can contain NUL bytes or unpaired surrogates (for example when a task emits binary data — we hit this in production where a run's error contained a ZIP/`PK\x03\x04...` payload). Postgres `jsonb` rejects these, so the triggerer crashes when it writes the event: ``` sqlalchemy.exc.DataError: (psycopg2.errors.UntranslatableCharacter) unsupported Unicode escape sequence DETAIL: \u0000 cannot be converted to text. ``` Because the invalid event is retried, the triggerer enters a `CrashLoopBackOff`, which stalls **all** deferrable tasks on the deployment, not just the offending one. ## How A small `make_jsonb_safe()` helper regex-strips `\x00` and `\ud800-\udfff` from the error string (non-string values pass through unchanged). It is applied to the extracted error in both the sync and async helpers, so the emitted trigger event is always persistable. ## Tests Added unit tests in `providers/databricks/tests/unit/databricks/utils/test_databricks.py`: - direct `make_jsonb_safe` cases (NUL strip, unpaired-surrogate strip, clean-text passthrough, non-string passthrough) - `extract_failed_task_errors` and `extract_failed_task_errors_async` sanitize error output --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes - Cursor was used to help draft this change; the author reviewed it. Generated-by: Cursor Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
