lizthegrey opened a new issue, #1016: URL: https://github.com/apache/arrow-go/issues/1016
### Describe the bug, including details regarding any error messages, version, and platform. Heads-up about an upstream bug I introduced in `klauspost/compress`, which arrow-go picks up. I have no prior involvement with this project, so apologies for arriving cold — I am notifying the dependents I could identify as actually shipping the affected versions. **What it is** `klauspost/compress` **v1.19.0 and v1.19.1** contain generated arm64 assembly for the zstd decoder in which every function local is placed 8 bytes too low, so local 0 overwrites the **saved link register**. It affects four functions: `sequenceDecs_decode_arm64`, `sequenceDecs_decode_56_arm64`, `sequenceDecs_decodeSync_arm64`, `sequenceDecs_decodeSync_safe_arm64`. Root cause is an architecture difference the code generator did not account for: on x86 `CALL` pushes the return address above the frame, so local 0 sits at `SP+0`; on arm64 `BL` leaves it in the link register and the callee spills it to the *bottom* of its own frame, so `0(RSP)` is the saved LR and locals start at `8(RSP)`. **Scope — please read before prioritising** - **No data corruption.** Decoded output is always correct. Every local shifts by the same amount, so loads and stores agree — which is exactly why the upstream test suite and fuzzing never caught it. - **arm64 only.** amd64 is unaffected. - **Decompression only.** Compression does not touch these functions. - Manifests **only when something walks the stack** while one of those frames is live: GC stack scanning, panics, or CPU profiling. The likeliest trigger is a continuous profiler, since SIGPROF interrupts arbitrary code and unwinds immediately. The symptom would appear to be a Go runtime bug rather than a compression one — `fatal error: traceback stuck`, `runtime: unexpected return pc`, or a crash inside `runtime.gentraceback` / `runtime.scanobject` / `runtime.sigprof` — which is why I am flagging it rather than waiting for reports. **arrow-go's exposure** - `v18.7.0` requires `klauspost/compress v1.19.0` - HEAD requires `v1.19.1` - Both `parquet/compress/zstd.go` and `arrow/ipc/compression.go` use `zstd.NewReader`, so Parquet and IPC decode are both on the affected path **Suggested action** Nothing to change in arrow-go itself. The upstream fix is klauspost/compress#1176, currently awaiting review; when a patch release is cut, a dependency bump is sufficient. If anyone is running arm64 with continuous profiling and seeing unexplained runtime crashes, pinning `v1.18.x` is a workaround in the meantime. **Honest caveat** I have not reproduced a crash from this. The diagnosis is from the frame layout, verified by disassembling probe functions at multiple frame sizes; the failure mode is inferred from it rather than observed. If you have seen an arm64 traceback crash with Parquet or IPC zstd decode in the interrupted goroutine, a stack dump on the upstream PR would be valuable — it would turn a reasoned diagnosis into a confirmed one. ### Component(s) Parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
