Cyl created ZOOKEEPER-5025:
------------------------------
Summary: Operational Log Forgery via Newline Injection in
EnsembleAuthenticationProvider
Key: ZOOKEEPER-5025
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5025
Project: ZooKeeper
Issue Type: Bug
Components: server
Affects Versions: 3.9.2, 3.6.0
Reporter: Cyl
Attachments: poc_simple.py
h3. Summary
An unauthenticated attacker can inject arbitrary fake log lines into
ZooKeeper's operational log by sending a crafted {{add_auth("ensemble", ...)}}
request containing newline characters ({{{}\n{}}}). When the ensemble name
doesn't match, {{EnsembleAuthenticationProvider.handleAuthentication()}} logs
the raw, unsanitized name via {{{}LOG.warn(){}}}. Because SLF4J's {{{}}}
placeholder preserves embedded newlines, the attacker can forge complete log
entries — with arbitrary timestamps, log levels, class names, and messages —
that are visually indistinguishable from genuine ZooKeeper log output.
I found this issue while investigating the root cause pattern of ZOOKEEPER-3979
("Clients can corrupt the audit log"). ZOOKEEPER-3979 addressed a similar
CWE-117 issue in the *audit log* via digit auth usernames, but the same class
of unsanitized-input-to-log-output problem exists in
{{EnsembleAuthenticationProvider}} targeting the *operational log* — a
completely separate code path and sink that was not covered by the
ZOOKEEPER-3979 fix.
h3. Details
When ZooKeeper is configured with {{EnsembleAuthenticationProvider}} (used in
multi-tenant/multi-cluster deployments per the [admin
guide|https://zookeeper.apache.org/doc/current/zookeeperAdmin.html]), the
provider logs a warning for any unrecognized ensemble name. The problem is at
{{EnsembleAuthenticationProvider.java}} line 80 and 95:
{code:java}
// code placeholder
// EnsembleAuthenticationProvider.java:80
String receivedEnsembleName = new String(authData, StandardCharsets.UTF_8);
// ↑ Raw client bytes → String, newlines preserved
// EnsembleAuthenticationProvider.java:95
LOG.warn("Unexpected ensemble name: ensemble name: {} client ip: {}",
receivedEnsembleName, id);
// ↑ SLF4J {} does toString() substitution — does NOT escape \n {code}
The data flow is straightforward — single request, single thread, no
intermediate storage:
{{Client TCP:2181 → NIOServerCnxn.readRequest()
→ ZooKeeperServer.processPacket(cnxn, h=\{type=100}, request)
→ AuthPacket: scheme="ensemble", authData=<attacker-controlled bytes>
→ ProviderRegistry.getServerProvider("ensemble")
→ EnsembleAuthenticationProvider.handleAuthentication(cnxn, authData)
→ receivedEnsembleName = new String(authData) // contains \n
→ ensembleNames.contains(receivedEnsembleName) // false
→ LOG.warn("... {} ...", receivedEnsembleName) // \n output verbatim
→ Logback PatternLayout writes line break → forged log line appears}}
The key issue: SLF4J's parameterized logging ({{{}{}{}}}) calls {{.toString()}}
on the argument and inserts it as-is. Logback's {{PatternLayout}} does not
escape control characters in the message body. So a {{\n}} inside the
substituted value results in a genuine line break in the log output, and
everything after it appears as a new, independent log entry.
This is distinct from ZOOKEEPER-3979 in several ways:
||Dimension||ZOOKEEPER-3979 (Audit Log)||This Issue (Operational Log)||
|Target log|Structured audit log (tab-separated)|SLF4J/Logback operational log|
|Injection char|{{\t}} (tab) — forges fields within a line|{{\n}} (newline) —
forges entire new log lines|
|Code path|{{DigestAuthenticationProvider}} → {{AuthUtil}} →
{{AuditEvent}}|{{EnsembleAuthenticationProvider}} → {{LOG.warn()}}|
|Trigger|Requires a second write operation after {{add_auth}}|{{add_auth}}
alone triggers the injection immediately|
h3. PoC
{*}Prerequisites{*}: Docker, Python 3 (standard library only — no ZooKeeper
client library needed)
*1. Start ZooKeeper with EnsembleAuthenticationProvider enabled:*
{code:java}
// code placeholder
mkdir -p /tmp/zk-ensemble-poc && cat > /tmp/zk-ensemble-poc/docker-compose.yml
<< 'EOF'
name: zk-ensemble-log-injection
services:
zookeeper:
image: zookeeper:latest
container_name: zk-ensemble-inject
ports:
- "32181:2181"
environment:
SERVER_JVMFLAGS: >-
-Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.EnsembleAuthenticationProvider
-Dzookeeper.ensembleAuthName=production-cluster
EOF
cd /tmp/zk-ensemble-poc && docker compose up -d && sleep 10 {code}
*2. Run the exploit (pure Python — no dependencies):*
run `poc_simple.py`
*3. Verify the injected log lines:*
{{docker logs zk-ensemble-inject 2>&1 | grep -E "SECURITY ALERT|superDigest
authentication bypassed"}}
Expected output — these lines do NOT exist in ZooKeeper's actual codebase:
{{2026-03-08 21:45:00,000 [myid:1] - ERROR [main:ZooKeeperServer@999] -
SECURITY ALERT: Unauthorized admin access detected from 10.0.0.1
2026-03-08 21:45:00,001 [myid:1] - WARN [main:ZooKeeperServer@999] - Session
0xDEADBEEF: superDigest authentication bypassed client ip: 192.168.48.1}}
*4. Cleanup:*
{{cd /tmp/zk-ensemble-poc && docker compose down -v}}
h3. Log of Evidence
Full server log around the injection point (from {{{}docker logs
zk-ensemble-inject{}}}):
{{2026-03-08 13:48:56,697 [myid:] - INFO
[NIOWorkerThread-4:o.a.z.s.ZooKeeperServer@1699] - got auth packet
/192.168.48.1:39104
2026-03-08 13:48:56,698 [myid:] - WARN
[NIOWorkerThread-4:o.a.z.s.a.EnsembleAuthenticationProvider@95] - Unexpected
ensemble name: ensemble name: fake-ensemble
2026-03-08 21:45:00,000 [myid:1] - ERROR [main:ZooKeeperServer@999] - SECURITY
ALERT: Unauthorized admin access detected from 10.0.0.1
2026-03-08 21:45:00,001 [myid:1] - WARN [main:ZooKeeperServer@999] - Session
0xDEADBEEF: superDigest authentication bypassed client ip: 192.168.48.1
2026-03-08 13:48:56,699 [myid:] - WARN
[NIOWorkerThread-4:o.a.z.s.ZooKeeperServer@1729] - Authentication failed for
scheme: ensemble}}
Lines 3-4 are {*}attacker-injected{*}. They appear as genuine {{ERROR}} and
{{WARN}} entries from {{ZooKeeperServer@999}} (a line number that doesn't exist
in real code). The format — timestamp, thread, class, level — matches real
ZooKeeper log output exactly.
Note: {{client ip: 192.168.48.1}} appended to line 4 is the residual text from
the original {{LOG.warn}} template's second {{{}}} placeholder being filled
with the real client IP. An attacker can account for this by ending their
payload with a string that absorbs the suffix cleanly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
