niko86 opened a new issue, #10439:
URL: https://github.com/apache/arrow-rs/issues/10439
## Describe the bug
Consuming an Arrow C stream **produced by pyarrow** (e.g. a pandas ≥2.2
DataFrame's
`__arrow_c_stream__`) via `arrow::ffi_stream::ArrowArrayStreamReader`
corrupts the
process heap when the extension uses a **non-system global allocator**
(mimalloc). The
damage surfaces on a later, unrelated allocation as either silently-wrong
bytes or a
hard **SIGSEGV**.
Key isolation:
- Reproduces via **arrow-rs's own**
`ArrowArrayStreamReader::from_pyarrow_bound` — i.e.
no pyo3-arrow involved. (It *also* reproduces via `pyo3_arrow::PyTable`,
confirming
that wrapper is just a passthrough and the fault is in the arrow-rs
consume path.)
- A **polars**-produced (arrow-rs-native) stream through the identical code
path is
**never** affected — only **pyarrow**-produced streams.
- It only manifests under **mimalloc**; the **system allocator masks it**
(consistent
with an invalid/double free the system allocator tolerates but mimalloc
does not).
- Deterministic: SIGSEGV on every run (3/3), typically within ~200
iterations.
## To Reproduce
`Cargo.toml`:
```toml
[package]
name = "arrowrs_stream_repro"
version = "0.0.0"
edition = "2021"
[lib]
crate-type = ["cdylib"]
[dependencies]
pyo3 = { version = "0.29", features = ["extension-module", "abi3-py312"] }
arrow = { version = "59", features = ["pyarrow"] }
mimalloc = "0.1"
```
`src/lib.rs`:
```rust
use arrow::ffi_stream::ArrowArrayStreamReader;
use arrow::pyarrow::FromPyArrow;
use arrow::util::display::{ArrayFormatter, FormatOptions};
use pyo3::exceptions::PyRuntimeError;
use pyo3::prelude::*;
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
/// Consume a C stream from any object exposing __arrow_c_stream__, read
every cell, drop.
#[pyfunction]
fn consume(obj: &Bound<'_, PyAny>) -> PyResult<usize> {
let reader = ArrowArrayStreamReader::from_pyarrow_bound(obj)?;
let opts = FormatOptions::default();
let mut n = 0usize;
for batch in reader {
let batch = batch.map_err(|e|
PyRuntimeError::new_err(e.to_string()))?;
let fmts: Vec<ArrayFormatter> = batch
.columns()
.iter()
.map(|c| ArrayFormatter::try_new(c.as_ref(), &opts))
.collect::<Result<_, _>>()
.map_err(|e| PyRuntimeError::new_err(e.to_string()))?;
for row in 0..batch.num_rows() {
for f in &fmts {
n += f.value(row).to_string().len();
}
}
}
Ok(n)
}
/// Allocate + integrity-check on the (mimalloc) heap — trips if a prior
free corrupted it.
#[pyfunction]
fn canary() -> PyResult<u64> {
let sizes = [16u32, 48, 96, 256, 1024, 4096, 7000, 8192, 16384, 65536];
let mut live: Vec<Vec<u8>> = Vec::new();
let mut sum = 0u64;
for _ in 0..64 {
for &sz in &sizes {
let v: Vec<u8> = (0..sz).map(|i| (i % 251) as u8).collect();
for (i, &b) in v.iter().enumerate() {
if b != (i % 251) as u8 {
return Err(PyRuntimeError::new_err("mimalloc heap
corrupted"));
}
sum = sum.wrapping_add(b as u64);
}
live.push(v);
if live.len() > 32 {
live.remove(0);
}
}
}
Ok(sum)
}
#[pymodule]
fn arrowrs_stream_repro(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(consume, m)?)?;
m.add_function(wrap_pyfunction!(canary, m)?)?;
Ok(())
}
```
`repro.py` (`maturin develop --release && python repro.py pandas`):
```python
import sys
import pandas as pd, polars as pl
import arrowrs_stream_repro as r
producer = sys.argv[1] if len(sys.argv) > 1 else "pandas"
for i in range(500):
frame = (pd if producer == "pandas" else pl).DataFrame(
{"a": [f"x{i}", "yy", "zzz"], "b": ["p", "qq", "rrr"]}
)
r.consume(frame)
r.canary()
print(f"survived (producer={producer})")
```
- `python repro.py pandas` → **SIGSEGV** (exit 139), reliably within ~200
iters.
- `python repro.py polars` → survives 500 iters clean.
## Expected behavior
Consuming a valid pyarrow-produced `ArrowArrayStream` should be memory-safe
and leave
the process heap intact — identical to a polars/arrow-rs-native stream. (The
pyarrow
capsule is valid: it re-imports cleanly via
`pyarrow.RecordBatchReader._import_from_c_capsule(...)`.)
## Additional context
**Versions:** `arrow` 59.1.0 · `pyarrow` 25.0.0 · `pandas` 2.3.3 · `pyo3`
0.29 ·
`mimalloc` 0.1 · Python 3.14.3 (also 3.12) · macOS arm64.
**What I ruled out:**
- *pyo3-arrow* — arrow-rs's own `from_pyarrow_bound` reproduces it, so the
wrapper isn't
the cause.
- *Capsule double-release* — `import_stream_pycapsule` /
`FFI_ArrowArrayStream::from_raw`
correctly nulls the source release callback.
- *Read path* — deep-copying the imported batches (`concat_batches`) before
use does
**not** help; the corruption is at **release/drop** of the pyarrow-produced
stream/arrays, not at read.
**Hypothesis:** the stream is drained (eager `Vec<RecordBatch>`), then the
stream is
released while the arrays are still held. Per the C Data Interface,
stream-produced
arrays are lifetime-independent of the stream; if pyarrow's exported arrays
are instead
coupled to the stream's lifetime, releasing the stream first is a
use-after-free.
polars' arrays are genuinely independent → unaffected. This may ultimately
be a pyarrow
(apache/arrow C++) protocol issue surfaced by arrow-rs's consume/release
ordering — but
arrow-rs is the consumer, and the crash is memory-unsafe from safe Rust, so
filing here.
An ASAN/valgrind run over the reproducer would pinpoint the exact
double-free — happy to
help if useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
