MichielMAnalytics opened a new pull request, #76585:
URL: https://github.com/apache/airflow/pull/76585
<p align="center">
<img
src="https://raw.githubusercontent.com/MichielMAnalytics/airflow/07114b0f5128a8b9c4c0dd704da770ba3c6967cf/boxd-x-airflow.png";
alt="boxd × Apache Airflow" width="700">
</p>
> [!NOTE]
> **draft, please don't merge yet.** i'm going through this by hand first
and will mark it ready for review myself
I work on [boxd](https://boxd.sh/?ref=airflow). boxd runs full Linux
microVMs (KVM) with a persistent disk that start in about a second, and can be
hibernated, snapshotted and forked. This PR adds `BoxdSandboxBackend` next to
the Modal, Islo and Boat backends, so a `SandboxToolset` can run the agent's
commands in a boxd machine. Like Islo and Boat, the worker only talks to an
HTTPS/gRPC API, so it works from a containerized worker and on Kubernetes.
### What changed
- `sandbox/boxd.py`: `BoxdSandboxBackend`, an `AttachableSandboxBackend`.
The ownership tags are boxd machine labels, so a task can provision a machine
and an agent task can attach to it (Modal is the only other backend that
supports this today).
- Every sandbox is created as an *isolated* boxd machine (no metadata
endpoint, no in-machine CLI, no integration tokens, no private-network peers),
with `SandboxSpec.env` and the network policy in the same create request. The
network policy becomes the machine's egress allowlist from birth:
`block_network=True` is `none`, hostnames and IPv4 CIDRs map to allowlist
entries, and an allowlist next to `block_network=False` is refused, as on
Modal. After create, the backend checks that boxd reports the machine isolated
and its allowlist in force, and deletes it otherwise.
- Commands run through `bash` under GNU `timeout` with output captured to
scratch files in the guest, so only the tail of each stream comes back, a
timed-out command keeps the machine and its files, and a background process
does not hold the call. The machine is deleted (`sandbox_terminated`) only when
the API call itself outlives the in-guest deadline by 30s.
- `write_file` uses boxd's file API (creates parents, works for files far
over the command line limit). Reads, listings and exports keep the inherited
shell implementations, so every transfer is bounded by a command deadline and
`max_bytes` is enforced in the guest.
- Credentials: a generic connection (`boxd_conn_id`, password is the API
key) or `BOXD_API_KEY`. The key is required on purpose: the boxd SDK can also
sign in as the boxd machine it runs on, and a worker on a boxd machine must not
quietly act as that machine's owner.
- `delete_after` (default 3600s) is the server-side backstop and the
lifetime an attaching toolset is told about.
- The `boxd` SDK is the optional `[boxd]` extra (`boxd>=0.2.22`, the first
release with the file API and egress in a machine spec) and is imported lazily,
like Islo. Docs: a new `boxd (hosted microVM)` section in
`sandbox/backends.rst`, plus the backend lists, network table, attach section
and lifecycle table. New unit tests and a system test. No newsfragment, per the
provider rules.
### How it maps onto `SandboxBackend`
| Airflow | boxd |
|---|---|
| `create(spec)` | `machines.create(name, spec=<isolated, labels, env,
egress, size, snapshot>, delete_after=...)`, then `wait_until_ready` |
| `run_command` | `machines.exec(id, ["sh", "-c", wrapper, ...],
timeout=..., encoding="buffer")` |
| `write_file` | `machines.fs.write` |
| `read_file`, `list_directory`, `export_file` | inherited shell defaults
over `run_command` |
| `destroy` | `machines.delete` (not found is fine) |
| `read_tags` / `write_tags` | machine labels (`machines.get().labels`,
`update_labels(..., replace_all=True)`) |
| `SandboxSpec.block_network` / `allow_egress_to` / `allow_egress_to_cidrs`
| machine egress allowlist (`none`, hostnames, IPv4 CIDRs) |
| `SandboxSpec.owner`, expiry, network stamp | `airflow_owner`,
`airflow_expires_at`, `airflow_network` labels |
### Where boxd does not map cleanly
1. **Account environment variables reach the sandbox.** boxd sets the
organization's `all`-scope and the key owner's `private`-scope environment
variables and secrets in every machine, isolated ones included. That breaks
"only `SandboxSpec.env` enters the sandbox". I could not switch it off from the
API, so it is a warning in the docs (and in the limitations list on the sandbox
index page): use an org whose `all`/`private` scopes hold nothing model code
should see, or bind secrets to hosts so the machine only holds a placeholder.
2. **The allowlist is not in force at the very first moment.** In my tests a
machine created with an allowlist answered commands a moment before boxd
applied it, and the first HTTPS request in that window got through (1 of 1 in
two trials, first ~0.5s). boxd documents allowlist changes as live "within
about a second", so `create` waits 3 seconds before it hands over a machine
with an allowlist. After that I saw no leak in 6 of 6 fresh machines. I am
taking this up on the boxd side too.
3. **HTTP and HTTPS are admitted by hostname only.** A CIDR entry admits
other protocols (DNS to `1.1.1.1` worked), but HTTPS to a listed address is
refused unless its hostname is listed. That is stricter than the toolset's
"these address ranges, on any port", not looser, and the docs say to list the
hostname for a web API at a fixed address. In the other direction, plain HTTP
to a listed host works, while the model is told HTTPS only.
4. **Label limits.** A boxd label value holds at most 256 characters. An
owner or holder longer than that is refused with a clear error; a network
policy too long for a label is still enforced but not recorded, so an attaching
agent is told to test what it can reach.
5. **Labels are not a lock.** Same as on Modal: anyone with an API key of
the org can change them. The base class's read-write-read check applies
unchanged.
6. **Commands run as `boxd` with passwordless `sudo`.** The isolation and
egress rules are enforced outside the guest, so root inside does not lift them.
Documented.
### Test plan
Everything below ran on this branch. Live tests used a boxd API key fenced
to a test org, one machine at a time, and every machine was deleted afterwards
(checked with a list).
- **Unit tests** (`tests/unit/common/ai/sandbox/test_boxd.py`, fakes for the
SDK, plus the real command wrapper run through `sh`/`bash`/GNU `timeout` on the
host): **68 passed** with `boxd==0.2.22` (the locked version and the extra's
floor) via `uv run --project providers/common/ai pytest`, and 68 passed with
`boxd==0.3.0`, the current release (still inside the workspace's 4-day
`exclude-newer` window, which is why the lock has 0.2.22). The entry-point
checks below passed on both versions; the network/attach checks ran on 0.3.0,
the system test and agent runs on 0.2.22.
- **Sandbox and toolset suites** (`tests/unit/common/ai/sandbox`,
`toolsets/test_sandbox.py`) with uv: 1044 passed; the 14 errors are
`ModuleNotFoundError: airflow_shared` in tests I did not touch and come from
that environment (same error with my change stashed), so I ran the whole
provider unit suite in Breeze as well: `breeze run pytest
providers/common/ai/tests/unit --skip-db-tests -n 4` gave **4566 passed, 27
skipped**, and `--run-db-tests-only` gave **88 passed, 8 skipped**, no failures.
- **Static checks**: `prek run --stage pre-commit` on the changed files
passes (ruff, ruff-format, codespell, provider.yaml and `get_provider_info`
checks, license headers, docs checks), and `prek run mypy-providers --stage
pre-push` on the changed Python files passes.
- **Live, every entry point** (`create`, `run_command`, `write_file`,
`read_file`, `list_directory`, `export_file`, `destroy`,
`read_tags`/`write_tags`, `attach`/`release`): 31/31 checks, including env with
quotes, `bash` syntax, exit codes and stderr, a 3s timeout returning the
partial stdout and leaving no process behind, a background process not
blocking, a 200k-line output tail-bounded at 100 bytes, 3,000,000 bytes of
output intact, a 3 MiB binary round trip, an over-budget read reporting the
real size, missing path / directory / permission errors coming back as
recoverable, a symlink and a newline in a file name in `list_directory`, a
sha256-identical 3 MiB export, export of a relative path, and a command after
`destroy` failing terminally.
- **Live, network, credentials, attach**: 26/26 checks: no key and a wrong
key give clear terminal errors (no machine left behind), a bad size and a
missing snapshot are terminal, `size="1x4"` and `snapshot=` work (snapshot made
and deleted in the test), open egress reaches the internet, the default spec
reaches nothing (DNS refused), a hostname list admits only that host, a CIDR
admits DNS to that address, a private CIDR is refused at create, an owner
longer than 256 characters is refused, and attach/re-attach/second holder/wrong
owner/release behave as the base class says.
- **System test** `example_sandbox_toolset_boxd.py` (new; a scripted model
through every tool, the timeout keeping the machine, no egress, teardown
confirmed; and a second task that provisions with an owner, attaches two runs,
refuses a wrong owner and collects the file): `pytest --system` **1 passed**.
- **Real agent loops with Claude** (`anthropic:claude-opus-5-5`): a
pydantic-ai `Agent` with `SandboxToolset(BoxdSandboxBackend(...))` wrote and
ran a script, read its output, listed the directory and correctly reported that
the network was blocked (5 tool calls, machine deleted after the run). Then a
Dag run through `dag.test()` with two `AgentOperator`s: one attached to a
machine a `@task` provisioned (`attach_to="{{
ti.xcom_pull(task_ids='provision') }}"`) and wrote a revenue report the
collecting task read out, and one used `exports=` to copy a CSV the agent built
to `file://`. **5 of 5 tasks succeeded.**
- **Docs**: both Python snippets in the new section ran as written inside an
agent run (the second with `AIRFLOW_CONN_BOXD_DEFAULT` and no `BOXD_API_KEY`),
and `pip install "./providers/common/ai[boxd]"` into a fresh venv installs the
SDK and imports the backend.
- Not run: the docs build (`breeze build-docs`), and the full CI matrix
(other Python versions and databases).
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
![https://raw.githubusercontent.com/MichielMAnalytics/airflow/07114b0f5128a8b9c4c0dd704da770ba3c6967cf/boxd-x-airflow.png"](https://raw.githubusercontent.com/MichielMAnalytics/airflow/07114b0f5128a8b9c4c0dd704da770ba3c6967cf/boxd-x-airflow.png"){kind=link}