Awesome to see! I agree with Curt that I hope this breathes new life into the device interface. I'll take a look at the Arrow Go issues and see what I can do.
I might hold off on some of them for now in favor of them potentially being fodder for the hackathon at Community Over Code in October though. --Matt On Tue, Sep 8, 2026, 11:00 AM Curt Hagenlocher <[email protected]> wrote: > Very cool! Maybe this will breathe new life into the Device Data interface > ;). > > On Tue, Sep 8, 2026 at 12:43 AM PRATEEK SINGH > <[email protected]> wrote: > > > > Hello Arrow developers, > > > > ArrowMetal 0.1.0 is out: Apache Arrow compute kernels that run on Apple > silicon GPUs through Metal. > > Arrow buffers live in shared-storage Metal memory, so an array is > simultaneously a valid CPU Arrow > > buffer and a valid GPU buffer; nothing is uploaded or downloaded. Data > crosses through the C Data > > Interface, the C Stream Interface and the C Device Data Interface > (ARROW_DEVICE_METAL), so it works > > with pyarrow, Polars, DuckDB, pandas, arrow-rs, Arrow Go, Arrow JS, the > arrow R package and > > arrow-swift. > > > > Site and docs: https://arrowmetal.org > > Source (Apache-2.0): https://github.com/singhpratech/ArrowMetal > > Python: pip install arrowmetal (macOS 14+, Apple silicon) > > Rust: arrowmetal = "0.1.0" on crates.io > > > > What was measured, on an M4 Max, everything in the repository: > > > > - 307 of Arrow v25's 307 compute function names: 283 on the GPU, 17 on > the host, 7 with a stated > > limitation. The table is generated from a registry the test suite > executes against pyarrow.compute. > > - 39,069 differential cases against pyarrow over 45 column types, with > every divergence classified > > and documented; 769 Swift tests against a CPU oracle; 2,452 Python > cases. > > - 339 benchmark rows over 173 operations against Polars (lazy engine), > pyarrow (Acero), pandas and > > numpy, each at its most parallel idiom, with the cores each call used > recorded beside it: 145 rows > > at or above 3x, 102 between 1x and 3x, 77 where the CPU idiom is > ahead, 15 with no CPU > > equivalent. The 77 are grouped by measured cause, each with what would > change it. > > - Seven languages on one C ABI: Swift, Python, C, Rust, Go, TypeScript > and R, each binding with its > > own test suite against that language's Arrow library. > > > > Where it is ahead is on shape: gathers, multi-key and high-cardinality > group-by, sorts and string > > predicates over columns that stay on the GPU between calls, at a > fraction of the CPU time the same > > rows cost on the other side. A single memory-bound pass over one column > is a tie against twelve to > > fifteen cores, and under a million rows the dispatch is the operation; > both are on the site. > > > > Two things for this list in particular: > > > > 1. Findings in other projects. The differential matrix and the bindings > turned up seventeen > > divergences or limitations outside ArrowMetal, each with a test in > the repository, and thirteen > > of them are now with the projects: five pyarrow issues (#51222 > binary_slice's default stop > > overflowing, #51223 fill_null_forward/backward and replace_with_mask > reading the wrong bits on a > > sliced boolean array, #51224 winsorize ignoring the offset, #51225 > utf8_normalize never > > composing, #51227 R's Array$create leaving buffers at offset 48 mod > 64), a reproduction added > > to #36110 (DST after the tabulated transitions, with a fat TZif > file), evidence and a pull-request > > offer on #39793 (the R package's unexported C Data Interface > allocators), three arrow-go issues > > (#1296 no aggregates in the compute registry, #1297 allocator > alignment, #1298 > > NewDatumWithoutOwning), and comments on arrow-js#88 and > swiftlang/swift#90477. The tracker with > > where every report stands, updated twice a day from the projects' own > trackers, is > > https://arrowmetal.org/#upstream and docs/UPSTREAM.md in the > repository. Corrections welcome, > > there and here. > > > > 2. The C Device Data Interface. ArrowMetal exports ARROW_DEVICE_METAL > device arrays and imports them > > with a sync event; the C Device Stream export is on the roadmap. If > anyone is working on device > > interop on Apple platforms, or on nanoarrow's device extension for > Metal, I would like to > > compare notes. > > > > Everything the site claims is measured, and the rows it does not win are > published beside the ones > > it does. Questions, corrections and benchmark CSVs from other machines > are welcome on the tracker. > > > > Prateek Singh > > https://arrowmetal.org > > >
