Public bug reported:
Release: Ubuntu 26.04 LTS (resolute)
Packages: podman 5.7.0+ds2-3build1, golang-github-containers-buildah
1.42.1+ds1-2build1
Expected: PUT /containers/{id}/archive (and podman cp) copies the tar into the
container and returns 200.
Actual: the request returns 500 "passing bulk input to subprocess: write |1:
broken pipe" - every time when
the client streams the tar in chunks (as Testcontainers/Docker.DotNet does),
intermittently otherwise.
Also affects: podman (rebuild needed after the buildah source fix).
[Impact]
Copying a tar archive into a container through the Docker-compatible API
(PUT /containers/{id}/archive, as used by Testcontainers, docker-compose
tooling and `podman cp`) fails intermittently with:
level=error msg="passing bulk input to subprocess: write |1: broken pipe"
"PUT /v1.44/containers/<id>/archive?path=%2F HTTP/1.1" 500
The cause is a race in buildah's copier (compiled into podman from
golang-github-containers-buildah-dev): the `buildah-copier` child process stops
reading at the tar end-of-archive marker (two 512-byte zero blocks) and exits,
while the parent is still writing the archive's trailing zero padding into the
pipe. The parent's write then fails with EPIPE and the API returns 500. Whether
it triggers depends on timing — most visibly when several copies run
concurrently or the padding arrives in a later read than the end marker.
Upstream bug: https://github.com/containers/buildah/issues/6573
Upstream fix: https://github.com/containers/buildah/pull/6678
commit f5124c43db85f29adcd58822c5b51fe2fe17cea5
("copier: drain tar stream to prevent broken pipe errors")
The fix is in buildah v1.44.0 and podman v6.0.0+. It is NOT in buildah 1.42.x
(resolute) or 1.43.x (stonking), so no Ubuntu release currently has it.
Real-world effect: test suites using Testcontainers (e.g. Testcontainers.MongoDb
with a replica set, which copies an init script into each container before
start) fail at fixture setup on rootless podman; one failed copy takes down the
whole suite. We see this daily on a shared Ubuntu 26.04 development server
(several users, rootless podman, per-user podman.socket as DOCKER_HOST).
The fix is a 9-line addition in copier/copier.go, copierHandlerPut(): after the
tar reader returns io.EOF, drain the remaining bulk input:
if _, err := io.Copy(io.Discard, bulkReader); err != nil {
logrus.Debugf("error draining remaining data from tar stream: %v",
err)
}
It applies to 1.42.1 (the function is unchanged around it).
[Test Plan]
On Ubuntu 26.04 with podman 5.7.0+ds2-3build1, as a regular (rootless) user.
The reproducer uploads a small tar with chunked transfer encoding, sending the
end-of-archive marker and the trailing padding in separate chunks — this is
what streaming clients such as Docker.DotNet (Testcontainers) do.
systemctl --user enable --now podman.socket
cat > /tmp/chunkput.py <<'EOF'
import socket, sys, io, tarfile, time
sock, cid = sys.argv[1], sys.argv[2]
b = io.BytesIO()
with tarfile.open(fileobj=b, mode="w") as tf: # 10240 bytes, padded
tf.addfile(ti, io.BytesIO(d))
data = b.getvalue()
s = socket.socket(socket.AF_UNIX); s.connect(sock)
s.sendall((f"PUT /v1.44/containers/{cid}/archive?path=%2Ftmp HTTP/1.1\r\n"
"Host: d\r\nContent-Type: application/x-tar\r\n"
"Transfer-Encoding: chunked\r\n\r\n").encode())
def chunk(x): s.sendall(f"{len(x):x}\r\n".encode() + x + b"\r\n")
chunk(data[:2048]); time.sleep(0.2); chunk(data[2048:]);
s.sendall(b"0\r\n\r\n")
print(s.recv(200).decode().split("\r\n")[0])
EOF
for i in 1 2 3 4 5 6; do
id=$(podman create docker.io/library/alpine:latest)
python3 /tmp/chunkput.py $XDG_RUNTIME_DIR/podman/podman.sock $id
podman rm -f $id >/dev/null
done
Before the fix: every request returns "HTTP/1.1 500 Internal Server Error" and
the podman.service journal shows "passing bulk input to subprocess: write |1:
broken pipe". Measured on our host: 6/6 failed. (Sending the same tar in a
single chunk fails intermittently, 3/6 in our run; many concurrent uploads
fail more often.)
After the fix: every request returns "HTTP/1.1 200 OK". Also check
`podman cp <file> <ctr>:/tmp/` and that the copied content is intact
(`podman exec <ctr> cat ...`).
[Where problems could occur]
The change only adds a read of the remaining input after a successful PUT
extraction, discarding bytes the tar reader already treats as end-of-archive
padding. A regression would show as a copy that waits until the sender closes
its side of the pipe; the sender always closes after writing the request body,
so this cannot block indefinitely. Errors from the drain are logged at debug
level and do not change the result. GET (copy-out) paths are untouched.
[Other Info]
- Package versions: podman 5.7.0+ds2-3build1, golang-github-containers-buildah
1.42.1+ds1-2build1, conmon 2.1.13, crun 1.23.1 (upstream static build;
Ubuntu's 1.21 is also
installed — the copier path does not involve the OCI runtime), kernel
7.0.0-31-generic,
cgroup v2, overlay storage via fuse-overlayfs.
- Because Ubuntu's podman is built against golang-github-containers-buildah-dev
rather than its own vendored copy, the patch belongs in
golang-github-containers-buildah and podman needs a rebuild to pick it up
(the buildah binary benefits too).
- Unrelated to the race, but seen while isolating it: a tar entry whose uid/gid
is outside the caller's subordinate ID range returns the same 500 message.
That is expected behaviour, not part of this report.
- Verified against the sources: upstream buildah v1.42.1's copierHandlerPut()
carries the exact context the upstream patch applies to, and Ubuntu's podman
source Build-Depends on golang-github-containers-buildah-dev (>= 1.42), with
no
vendored copy of the copier.
[Disclosure]
This investigation and this report were produced with the help of an AI
assistant (Anthropic's Claude, via Claude Code), working on our server under my
direction. It ran the diagnosis and wrote the draft. I reviewed the report
before filing. The reproducer in [Test Plan] was run exactly as written on our
Ubuntu 26.04 host and failed 6/6. The upstream references and the
package/source checks above were confirmed against the actual repositories, not
inferred.
** Affects: golang-github-containers-buildah (Ubuntu)
Importance: Undecided
Status: New
** Tags: resolute
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2170079
Title:
podman API "PUT /containers/{id}/archive" fails with 500 "passing bulk
input to subprocess: write |1: broken pipe" (buildah copier race,
upstream buildah#6573)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/golang-github-containers-buildah/+bug/2170079/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
