psvri opened a new issue, #7618: URL: https://github.com/apache/arrow-rs/issues/7618
Hello , Will the team be interested in taking over the project https://github.com/psvri/arrow-gpu . This is a gpu implementation of apache arrow written in rust using [wgpu](https://github.com/gfx-rs/wgpu) .Honestly I am not able to spend more time on it due to personal life and at the same time I dont want to make the project go to waste. My hope is, if its moved to the apache foundation there will be a lot more people who would contribute and make it successful. Its around 12k lines of rust code and another few k lines of wgsl shaders. I have implemented quite a lot of kernels and datatypes . For a list of supported kernels and dataypes you can check this [doc](https://psvri.github.io/arrow-gpu/) . At this moment its not as performant as I would like it be, but its certainly way faster than CPU kernels. For e.g to incr 10 million non null f32 arrow arrays on my laptop takes 25% of the time compared to the cpu arrow-rs implementation (This was my first time writing gpu kernels , so perhaps someone more experienced can improve it further). Benchmark code is available [here](https://github.com/psvri/arrow-gpu/blob/main/crates/benchmarks/benches/compare_gpu_arrow.rs) . ``` gpu f32 time: [2.4419 ms 2.4573 ms 2.4732 ms] cpu f32 time: [9.7053 ms 9.7598 ms 9.8199 ms] ``` The primary reason I had created this project was to get a cross platform, vendor independent gpu version of arrow. After researching and tinkering I landed on wgpu since thats the best lowest common denominator across platforms and vendors, and wgpu has very good chance of not being unmaintained given that its gonna be in the web . If time had permitted I would have liked to integrate this in other projects like pyarrow, datafusion, polars and apache-spark and even aligned it with https://arrow.apache.org/docs/format/CDeviceDataInterface.html . I am willing to help with all the efforts needed for the movement. Sincerely, psvri -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
