mrproliu opened a new pull request, #209:
URL: https://github.com/apache/skywalking-rover/pull/209
## Problem
On newer kernels (reproduced on GKE COS `6.12` / amd64), the access log
module fails to start and the pod enters `CrashLoopBackOff`:
```
Error: start module access_log failure: field GoTlsReadRet: program
go_tls_read_ret:
load program: argument list too long: 2367: (b7) r1 = 2048:
BPF program is too large. Processed 1000001 insn (truncated, 449 line(s)
omitted)
```
Despite the `argument list too long` wording, this is **not** an arguments
problem — it is the eBPF **verifier complexity limit**. The verifier processed
more than 1,000,000 instructions (`Processed 1000001 insn`) without converging
and rejected the program.
## Root cause
`go_tls_read_ret` fully `__always_inline`-expands the entire data path into
a single program:
- `process_write_data`
- → `read_socket_data` + `analyze_protocol` (HTTP/2 inference with nested
unrolled loops)
- → `get_or_create_active_conn`
- → `upload_socket_data` → `upload_socket_data_buf` (6× unrolled) **and**
`upload_socket_data_iov` (6× unrolled), each inlining
`__upload_socket_data_with_buffer`
The verifier explores both the buf and iov branches × 6 chunks × per-chunk
branches, multiplied by the protocol-analysis branches. This produces a
combinatorial path explosion. `go_tls` trips the limit first because it carries
the extra arch-specific `get_goid` / `go_regabi_regs` register reads on top of
the shared path; the other SSL probes (`openssl`, `node_tls`) walk the same
data path and are next in line.
## Fix
Convert the heaviest **shared** inline helpers on the data path into
**BPF-to-BPF sub-program calls** (`__noinline`). A non-inlined sub-program is
verified **once** instead of being re-verified for every inlined copy on every
code path, which sharply lowers the processed-instruction count.
Functions converted (all ≤ 5 args, so they fit the BPF calling convention
with no signature changes):
| File | Function |
|------|----------|
| `bpf/include/api.h` | new `__noinline` macro (guarded with `#ifndef`) |
| `bpf/include/socket_data.h` | `upload_socket_data_buf`,
`upload_socket_data_iov` |
| `bpf/include/protocol_analyzer.h` | `analyze_protocol` |
| `bpf/include/socket_reader.h` | `read_socket_data` |
| `bpf/accesslog/common/connection.h` | `get_or_create_active_conn` |
Because these live in shared headers, every program on the data path
benefits at once (`go_tls`, `openssl`, `node_tls`, syscall transfer).
This is purely an inlining-strategy change — **no maps, structs, or upload
semantics are modified**, so runtime behavior is unchanged.
## Compatibility
BPF-to-BPF calls require kernel **≥ 4.16**. The access log / network
monitoring path already requires 4.16+ (see `docs/en/setup/overview.md`), so
the supported-kernel matrix is unchanged.
## Verification
Built the amd64 object and loaded `go_tls_read_ret` through the real kernel
verifier on amd64 (kernel `6.1`, same architecture and verifier generation as
the failing GKE node):
| Build | Processed insns | Result |
|-------|----------------:|--------|
| Before (original GKE error, amd64 / 6.12) | 1,000,001 | ❌ rejected (> 1M
limit) |
| After (this PR, amd64 / 6.1) | 178,670 | ✅ verified |
The program now verifies at roughly 1/6 of the instruction limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
