uranusjr opened a new pull request, #77281: URL: https://github.com/apache/airflow/pull/77281
Python, Go, Java, and TypeScript all have various differences in handling null values. This aligns the behavior between different runtimes as much as possible around both explicitly setting an XCom, and implicitly pushing a value by returning it from the task function. ### General guidelines * Setting an XCom to null explicitly is allowed, as designed in both the XCom database table, and the Execution API endpoint. * Whether a return value should be pushed is decided by the task function's signature. ### Python The implementation is already almost the best it can be, deviating only due to Python’s natural restrictions (no distinction between “no return value” and “return None”). ### Go The explicit `PushXCom` already fits the guideline. Two changes to return value handling: 1. A task function with return type `(*T, error)` now always pushes its result. `return nil, nil` pushes null (previous no value would be pushed). Unchanged: No value is pushed when the function’s return type is `error`. 2. If the task function's return type is `([]T, error)` or `(map[K]V, error)`, `return nil, nil` would push an empty array or map (previously no value would be pushed). This is due to Go commonly uses nil to represent an empty array or map. Pushing `nil` instead would make interfacing with such a task more difficult. (This is recommended by Claude, but I’m not entirely sure about it, ahving little experience in Go. Advices are welcome.) ### Java The `setXCom` function needs to be amended to accept and handle null. Before this, runtime would actually crash if null is returned from a task function. ### TypeScript Already doing things correctly. ### Documentation A line is added to each SDK's documentation to clarify the behavior. (Except Python, which already does.) ### Drive-by improvement XCom endpoints in the Execution API contain some stale handling carried over from SQLAlchemy 1.x no longer needed since we now require 2+. ### Next work This covers the _write_ side of the story, and should be enough to unblock task mapping design. However, there’s still an issue on the _read_ side. No SDK (Python included) currently has a way to distinguish between _no XCom_ and _null written to XCom_. Python is maybe the easiest to amend, but other SDKs would need further fixes in supervisor comms. Once we fix the read side interface, we could also add a cute feature to Python that mimics typed function returns in other SDKs. Python never distinguishes `return` and `return None` _at runtime_, but type checkers (Mypy etc.) do based on the function's return type annotation. ```python # These do not write an XCom. Same as current behavior. # Mypy accepts both 'return' and 'return None' in these cases. def no_annotation(): def explicit_return_type_none() -> None: def return_any() -> Any: # Bare return fails type checks, only 'return None' satisfies Mypy. # So we write a null XCom. def return_optional() -> T | None: # Both bare return and 'return None' would fail type checks. # But if the function ends with a None at runtime, a null XCom makes sense? def return_optional() -> T: ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
