jyericlin opened a new issue, #2874:
URL: https://github.com/apache/arrow-rs/issues/2874
**Describe the bug**
When creating a ArrayData object with Buffer that uses shared memory, it
seems there is a memory freed error when the ArrayData object is being dropped.
I think the reason is Rust is trying to free the pointer that is pointing to
the shared memory region which is not allowed.
Error:
```
data: ArrayData { data_type: UInt8, len: 4, null_count: 0, offset: 0,
buffers: [Buffer { data: Bytes { ptr: 0x1018c2000, len: 4, data: [0, 0, 0, 0]
}, offset: 0, length: 4 }], child_data: [], null_bitmap: None }
rust_playground(34082,0x10aa78600) malloc: *** error for object 0x1018c2000:
pointer being freed was not allocated
rust_playground(34082,0x10aa78600) malloc: *** set a breakpoint in
malloc_error_break to debug
```
**To Reproduce**
The following code can be use to reproduce the error.
```rust
use std::ptr::NonNull;
use shared_memory::{ShmemConf, ShmemError};
use arrow::{array::{ArrayData}, datatypes::DataType, buffer::Buffer};
use tracing::error;
fn main(){
let os_id = "test_200";
let shmem = match ShmemConf::new().size(1024).os_id(os_id).create() {
Ok(m) => m,
Err(ShmemError::MappingIdExists) =>
ShmemConf::new().os_id(os_id).open().unwrap(),
Err(e) => {error!("Unable to create or open shmem os_id {} :
{}",os_id, e);
return
}
};
let ptr = shmem.as_ptr();
let data = unsafe{
let buf = Buffer::from_raw_parts(NonNull::new(ptr).unwrap(), 4,
1024);
let data = ArrayData::builder(DataType::UInt8)
.len(4)
.add_buffer(buf)
.build()
.unwrap();
data
};
println!("data: {:?}", data);
//let _ = std::mem::ManuallyDrop::new(data);
}
```
Version:
arrow = "24.0.0"
shared_memory = "0.12.4"
**Analysis and Thought**
Adding ManuallyDrop (shown below) seems to remove the error, but I suspect
it would cause some sort of memory leak.
```let _ = std::mem::ManuallyDrop::new(data);```
Maybe the ArrayData struct can
1. Have flags to support whether one of its ```buffers: Vec<Buffer>```
should be dropped or not, or
2. Allow removal of buffer from its ```buffers: Vec<Buffer>```.
Not sure if this is considered a bug or a feature request, but I would like
to see if there is solution to this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
