diogosilva30 opened a new pull request, #77347:
URL: https://github.com/apache/airflow/pull/77347

   `XComObjectStorageBackend._write_large_value` writes with 
`path.open(mode="wb", compression=...)`. For a compressed write fsspec returns 
the compression wrapper (e.g. `GzipFile`) around the real file, and closing the 
wrapper does not close the file it wraps. The final upload is therefore left to 
the wrapped file's finalizer instead of happening when the `with` block exits.
   
   That used to be harmless. With gcsfs >= 2026.10 (`GCSFile.__del__` defers 
the close to a daemon "gcsfs-deferred-close" thread) it is not: a task process 
that pushes its last XCom and exits right away (the supervisor's forked child 
ends with `os._exit`) can exit before the deferred upload runs, so the object 
is never written. The XCom row, which is stored by the API server from the path 
the worker reports, then points to an object that does not exist, and 
downstream tasks fail with `FileNotFoundError` swallowed by `deserialize_value` 
into the raw path string (`'str' object has no attribute 'get'`, 
`NoneType`/kafka errors, etc.).
   
   I saw this on Airflow 3.4.0 with `xcom_objectstorage_compression = gzip` and 
a GCS bucket: A/B runs of simulated task children (write 35 compressed objects, 
`os._exit(0)`) lost the last object(s) only with gcsfs 2026.10.1, and not with 
2026.8.1 or with compression disabled.
   
   The fix opens the raw file and the compression wrapper in an `ExitStack`, so 
the wrapper is closed first and the raw file is then closed synchronously, 
which completes the upload before `_write_large_value` returns.
   
   The new test fakes `ObjectStoragePath.open` the way fsspec does for 
compression (wrapper leaves the raw file open) and asserts the raw file is 
closed and holds the full payload. It fails on the previous code for gzip and 
passes now; the uncompressed case is a control.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (Claude Code)
   
   Generated-by: Claude Code following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to