dheerajturaga opened a new pull request, #72546:
URL: https://github.com/apache/airflow/pull/72546
A Bash task can only produce one XCom today: the last line written to
stdout. Anything richer means serialising to JSON on a single line and parsing
it back with `output_processor`, which drags shell authors into quote-escaping
for even simple key/value pairs. The `KubernetesPodOperator` sidecar has the
same JSON contract, and every example of it in this repo writes a JSON
*literal* rather than a shell variable — which is telling.
Separately, the task runner only pushes XCom on the success path
(`_push_xcom_if_needed`), so a failing command cannot hand any diagnostic data
to downstream tasks or to the UI — precisely when that data is most wanted.
### What this adds
When `do_xcom_push` is set, `BashOperator` exposes a directory as
`$AIRFLOW_XCOM_DIR` plus an `xcom push` helper on `PATH`, and reads the
directory back in a `finally` block — so entries are pushed whether the command
succeeded, failed, or skipped.
Two rules govern what becomes an XCom:
* a **file's name is the key**, and its content (one trailing newline
stripped) is the value;
* a **`.json` suffix** parses the content and drops the suffix from the key.
```bash
echo "$rows" > "$AIRFLOW_XCOM_DIR/row_count" # no escaping, whatever $rows
contains
generate_report | xcom push report # stdin, for large or
multiline values
xcom push --json summary '{"rows": 42}' # same as writing summary.json
```
Because the filename carries the key, there is no format to escape — the
value can contain quotes, spaces, or newlines and still round-trips. The helper
is sugar over the documented directory contract, so scripts that reset `PATH`
or exec into a container can just redirect into `$AIRFLOW_XCOM_DIR`.
If the script writes a `return_value` entry and the command succeeds, that
value is returned from `execute()` rather than the stdout last line, so it
flows through the runner's normal return-value push and composes with
`multiple_outputs=True`.
### Notes for reviewers
* All changes are inside `providers/standard` — no SDK, scheduler, or
Execution API change. `ti.xcom_push` already writes through immediately, and
the `set_xcom` route does not gate on TI state.
* Values are **strings** unless `.json` is used; `row_count` is `"42"`, not
`42`. Documented explicitly rather than coercing digit-looking values.
* Collection errors (malformed JSON, oversized file, subdirectory) are
gathered rather than raised inline, and are downgraded to a warning when the
command already failed — a bad XCom file must never mask the original Bash
failure.
* This covers command failure, not worker death: a `SIGKILL` of the task
process skips the `finally`.
* Two new params: `xcom_helper_name` (`None` disables the helper) and
`max_xcom_file_size`.
### Deliberately left out
Nesting values via subdirectories. A `.json` file already covers structured
values, and nesting drags in a depth cap and a larger surface for a narrow use
case. It can be added in a later release without breaking anyone; removing it
once released could not. Subdirectories currently raise a clear error pointing
at `.json`.
### Testing
24 new tests in `TestBashOperatorXComDir` driving real bash, covering both
the success and failure paths, the helper, the error cases, and the
`return_value` override. `providers/standard` bash operator, decorator, sensor,
and subprocess-hook suites pass (121), as does the OpenLineage bash extractor.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5)
Generated-by: Claude Code (Opus 5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01SPiZuBWUwhduTb3JdNYzsN
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
