edgarriba opened a new issue, #6362: URL: https://github.com/apache/arrow-rs/issues/6362
hi, I have the [kornia-rs](https://github.com/kornia/kornia-rs) crate where I implemented a custom [`TensorStorage`](https://github.com/kornia/kornia-rs/blob/main/crates/kornia-core/src/storage.rs#L28) struct out of `ScalarBuffer` and I have some issues while extending a method `TensorStorage::from_vec`. This struct has the particularity that holds an Allocator which the intention is that the user can specify, e.g for CpuTensor or CudaTensor (or more advanced devices/backends). Initally, I had implemented this using the [`Buffer::from_custom_allocation`](https://docs.rs/arrow-buffer/latest/arrow_buffer/buffer/struct.Buffer.html#method.from_custom_allocation) method aiming to create the storage with a zero-copy cost. However, I recently discovered after adding some tests that something fishy is going on with the memory pointers of the vector. I can quickly illustrate what I have and what I'm trying to figure out (https://github.com/kornia/kornia-rs/pull/125) ```rust pub struct TensorStorage<T, A: TensorAllocator> where T: SafeTensorType, { /// The buffer containing the tensor storage. data: ScalarBuffer<T>, /// The allocator used to allocate the tensor storage. alloc: A, } impl<T, A: TensorAllocator> TensorStorage<T, A> where T: SafeTensorType + Clone, { pub fn from_vec(vec: Vec<T>, alloc: A) -> Self { // NOTE: this is a temporary solution until we have a custom allocator for the buffer // create immutable buffer from vec //let buffer = unsafe { // // SAFETY: `vec` is properly aligned and has the correct length. // Buffer::from_custom_allocation( // NonNull::new_unchecked(vec.as_ptr() as *mut u8), // vec.len() * std::mem::size_of::<T>(), // Arc::new(vec), // ) //}; // create immutable buffer from vec // NOTE: this is a temporary solution until we have a custom allocator for the buffer let buffer = Buffer::from_vec(vec); // create tensor storage Self { data: buffer.into(), alloc, } } } ``` Then the crashing test ```rust #[test] fn test_tensor_storage_into_vec() { let allocator = CpuAllocator; let original_vec = vec![1, 2, 3, 4, 5]; let original_vec_ptr = original_vec.as_ptr(); let original_vec_capacity = original_vec.capacity(); let storage = TensorStorage::<i32, _>::from_vec(original_vec, allocator); // Convert the storage back to a vector let result_vec = storage.into_vec(); // check NO copy assert_eq!(result_vec.capacity(), original_vec_capacity); // THIS TEST IS FAILING assert!(std::ptr::eq(result_vec.as_ptr(), original_vec_ptr)); } ``` Thanks in advance! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
