I would support deprecation. I've never seen legitimate use of these messages, only confusion.
On Wed, Jul 1, 2026 at 9:25 AM Rok Mihevc <[email protected]> wrote: > I would support removing Tensor and SparseTensors from IPC > because FixedShapeTensorArray and VariableShapeTensorArray > seem like a better way forward and can be made to provide the > same functionality. > Also securing IPC messages is quite difficult given complexity of > tensor data structures especially once one factors in that every > IPC implementation needs to think about this. > > It would of course also be good to do some research downstream > to see if we have downsream users who depend on Tensor via IPC. > > Rok > > On Wed, Jul 1, 2026 at 5:59 PM Antoine Pitrou <[email protected]> wrote: > > > > > Hello, > > > > While the Arrow IPC format is primarily meant for RecordBatch transport, > > it also has dedicated messages for transporting N-dimensional Tensor and > > SparseTensor data. > > > > The Tensor message was added in 2017 > > (https://github.com/apache/arrow/pull/435), while the SparseTensor > > message was added in 2019 (https://github.com/apache/arrow/pull/3651). > > > > Both have not had any significant additions since 2020. Furthermore: > > > > * They are still marked experimental. > > * They are not exercised by the the cross-implementation integration > > test suite. > > * Convenience IPC reading APIs are generally architected around > > RecordBatch iteration, requiring more manual plumbing to read Tensor and > > SparseTensor messages from an IPC stream. > > * They are generally not reachable from Flight RPC APIs, or other > > high-level APIs such the C++ Dataset API. > > * A code search on GitHub reveals no apparent third-party usage of these > > message types. > > > > Nowadays, Arrow has canonical extension types for fixed-shape and > > variable-shaped tensors which allow passing (dense) N-dimensional > > tensors as regular RecordBatch columns. We could also add canonical > > extension types for the different kinds of sparse tensors, if there is > > demand for it. > > > > Despite not seeing much usage, these message types carry their > > maintenance baggage. Recently, we've had a number of security reports > > around the lack of validation of untrusted Tensor and SparseTensor > > messages in Arrow C++, requiring core developer intervention to analyze > > and fix those issues. > > > > I would like to suggest that we start deprecating these messages types > > in the IPC format, and that we recommend deprecating the corresponding > > APIs in all major Arrow implementations. What do you think? > > > > Regards > > > > Antoine. > > > > >
