diogosilva30 opened a new pull request, #77347: URL: https://github.com/apache/airflow/pull/77347
`XComObjectStorageBackend._write_large_value` writes with `path.open(mode="wb", compression=...)`. For a compressed write fsspec returns the compression wrapper (e.g. `GzipFile`) around the real file, and closing the wrapper does not close the file it wraps. The final upload is therefore left to the wrapped file's finalizer instead of happening when the `with` block exits. That used to be harmless. With gcsfs >= 2026.10 (`GCSFile.__del__` defers the close to a daemon "gcsfs-deferred-close" thread) it is not: a task process that pushes its last XCom and exits right away (the supervisor's forked child ends with `os._exit`) can exit before the deferred upload runs, so the object is never written. The XCom row, which is stored by the API server from the path the worker reports, then points to an object that does not exist, and downstream tasks fail with `FileNotFoundError` swallowed by `deserialize_value` into the raw path string (`'str' object has no attribute 'get'`, `NoneType`/kafka errors, etc.). I saw this on Airflow 3.4.0 with `xcom_objectstorage_compression = gzip` and a GCS bucket: A/B runs of simulated task children (write 35 compressed objects, `os._exit(0)`) lost the last object(s) only with gcsfs 2026.10.1, and not with 2026.8.1 or with compression disabled. The fix opens the raw file and the compression wrapper in an `ExitStack`, so the wrapper is closed first and the raw file is then closed synchronously, which completes the upload before `_write_large_value` returns. The new test fakes `ObjectStoragePath.open` the way fsspec does for compression (wrapper leaves the raw file open) and asserts the raw file is closed and holds the full payload. It fails on the previous code for gzip and passes now; the uncompressed case is a control. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (Claude Code) Generated-by: Claude Code following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
