pitrou commented on code in PR #37797: URL: https://github.com/apache/arrow/pull/37797#discussion_r1345648166
########## docs/source/format/CDataInterface/PyCapsuleInterface.rst: ########## @@ -0,0 +1,402 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +============================= +The Arrow PyCapsule Interface +============================= + +Rationale +========= + +The :ref:`C data interface <c-data-interface>` and +:ref:`C stream interface <c-stream-interface>` allow moving Arrow data between +different implementations of Arrow. However, these interfaces don't specify how +Python libraries should expose these structs to other libraries. Prior to this, +many libraries simply provided export to PyArrow data structures, using the +``_import_from_c`` and ``_export_from_c`` methods. However, this always required +PyArrow to be installed. + +This interface allows any library to export Arrow data structures to other +libraries that understand the same protocol. + +Goals +----- + +* Standardize the `PyCapsule`_ objects that represent ``ArrowSchema``, ``ArrowArray``, + and ``ArrowArrayStream``. +* Define standard methods to get these capsule objects, so any Python objects that + support export via Arrow can be recognized by other libraries. + + +Non-goals +--------- + +* Standardizing what public APIs should be used for import. This is left up to + individual libraries. + + +Comparison to DataFrame Interchange Protocol +-------------------------------------------- + +`The DataFrame Interchange Protocol <https://data-apis.org/dataframe-protocol/latest/>`_ +is another protocol in Python that allows for the sharing of data between libraries. +This protocol is complementary to the DataFrame Interchange Protocol. Many of +the objects that implement this protocol will also implement the DataFrame +Interchange Protocol. + +This protocol is specific to Arrow-based data structures, while the DataFrame +Interchange Protocol allows non-Arrow data frames and arrays to be shared as well. +Because of this, these PyCapsules can support Arrow-specific features such as +nested columns. + +This protocol is also much more minimal than the DataFrame Interchange Protocol. +It just handles data export, rather than defining accessors for details like +number of rows or columns. + +In summary, if you are implementing this protocol, you should also consider +implementing the DataFrame Interchange Protocol. + + +Comparison to __arrow_array__ protocol +-------------------------------------- + +The ``__arrow_array__`` protocol is a dunder method that defines how PyArrow +should import an object as an Arrow array. Unlike this protocol, it does +specific to PyArrow and isn't used by other libraries. It is also limited to +arrays and does not support schemas, tabular structures, or streams. + +PyCapsule Standard +================== + +`PyCapsule`_ allows for a ``name`` to be associated with the capsule, allowing +consumers to verify that the capsule contains the expected data. To make sure +Arrow structs are recognized, the following names should be used: + +.. list-table:: + :widths: 25 25 + :header-rows: 1 + + * - C Interface Type + - PyCapsule Name + * - ArrowSchema + - ``arrow_schema`` + * - ArrowArray + - ``arrow_array`` + * - ArrowArrayStream + - ``arrow_array_stream`` + + +Lifetime Semantics +------------------ + +The exported PyCapsules should have a destructor that calls the +:ref:`release callback <c-data-interface-released>` +of the Arrow struct, if it is not already null. This prevents a memory leak in +case the capsule was never passed to another consumer. + +If the capsule has been passed to a consumer, the consumer should have moved +the data and marked the release callback as null, so there isn’t a risk of +releasing data the consumer is using. +:ref:`Read more in the C Data Interface specification <c-data-interface-released>`. + +Just like in the C Data Interface, the PyCapsule objects defined here can only +be consumed once. + + +Export Protocol +=============== + +The interface is three separate protocols: + +* ``ArrowSchemaExportable``, which defines the ``__arrow_c_schema__`` method. +* ``ArrowArrayExportable``, which defines the ``__arrow_c_array__``. +* ``ArrowStreamExportable``, which defines the ``__arrow_c_stream__``. + +The protocols are defined below in terms of ``typing.Protocol``. These may be +copied into a library for the purposes of static type checking, but this is not +required to implement the protocol. + + +.. code-block:: python + + from typing import Tuple, Protocol + from typing_extensions import Self + + class ArrowSchemaExportable(Protocol): + def __arrow_c_schema__(self) -> object: + """ + Get a PyCapsule containing a C ArrowSchema representation of the object. + + The capsule will have a name of "arrow_schema". + """ + ... + + class ArrowArrayExportable(Protocol): + def __arrow_c_array__( + self, + requested_schema: object | None = None + ) -> Tuple[object, object]: + """ + Export array as a pair of PyCapsules for the ArrowSchema and ArrowArray. + + The ArrowArray capsule will have a name of "arrow_array". + + If requested_schema is passed, the callee should attempt to provide the + data in the requested schema. However, this is best-effort, and the + callee may return a PyCapsule containing an ArrowArray with a different + schema. This parameter is useful for cases where the underlying data + could be represented in multiple ways, and the caller has a preference + for how it is represented. For example, some systems have a single + integer type, but Arrow has multiple integer types with different + sizes and sign. + + Parameters + ---------- + requested_schema : PyCapsule | None + A PyCapsule containing a C ArrowSchema representation of a requested + schema. Conversion to this schema is best-effort. + + Returns + ------- + Tuple[PyCapsule, PyCapsule] + A pair of PyCapsules containing a C ArrowSchema and ArrowArray, + respectively. + """ + ... + + + class ArrowStreamExportable(Protocol): + def __arrow_c_stream__(self, requested_schema: object | None = None) -> object: + """ + Get a PyCapsule containing a C ArrowArrayStream representation of the object. + + The capsule will have a name of "arrow_array_stream". + + If requested_schema is passed, the callee should attempt to provide the + data in the requested schema. However, this is best-effort, and the + callee may return a PyCapsule containing an ArrowArray with a different + schema. This parameter is useful for cases where the underlying data + could be represented in multiple ways, and the caller has a preference + for how it is represented. For example, some systems have a single + integer type, but Arrow has multiple integer types with different + sizes and sign. + + Parameters + ---------- + requested_schema : PyCapsule | None + A PyCapsule containing a C ArrowSchema representation of a requested + schema. Conversion to this schema is best-effort. + + Returns + ------- + PyCapsule + A PyCapsule containing a C ArrowArrayStream representation of the + object. + """ + ... + +Schema Requests +--------------- + +In some cases, there might be multiple possible Arrow representations of the +same data. For example, a library might have a single integer type, but Arrow +has multiple integer types with different sizes and sign. As another example, +Arrow has several possible encodings for an array of strings: 32-bit offsets, +64-bit offsets, string view, and dictionary-encoded. A sequence of strings could +export to any one of these Arrow representations. + +In order to allow the caller to request a specific representation, the +``__arrow_c_array__`` and ``__arrow_c_stream__`` methods may take an optional +``requested_schema`` parameter. This parameter is a PyCapsule containing an +``ArrowSchema``. + +The callee should attempt to provide the data in the requested schema. However, +if the callee cannot provide the data in the requested schema, they may return +with the same schema as if ``None`` were passed to ``requested_schema``. + +If the caller requests a schema that is not compatible with the data, +say requesting a schema with a different number of fields, the callee should +raise an exception. The requested schema mechanism is only meant to negotiate +between different representations of the same data and not to allow arbitrary +schema transformations. + + +.. _PyCapsule: https://docs.python.org/3/c-api/capsule.html + + +Examples +======== + +Create a PyCapsule +------------------ + +To create a PyCapsule, use the `PyCapsule_New <https://docs.python.org/3/c-api/capsule.html#c.PyCapsule_New>`_ +function. The function must be passed a destructor function that will be called +to release the data the capsule points to. It must first call the release +callback if it is not null, then free the struct. + +Below is the code to create a PyCapsule for an ``ArrowSchema``. The code for +``ArrowArray`` and ``ArrowArrayStream`` is similar. + +.. tab-set:: + + .. tab-item:: C + + .. code-block:: c + + #include <Python.h> + + void ReleaseArrowSchemaPyCapsule(PyObject* capsule) { + ArrowSchema* schema = + (ArrowSchema*)PyCapsule_GetPointer(capsule, "arrow_schema"); + if (schema->release != NULL) { + schema->release(schema); + } + free(schema); + } + + PyObject* MakeArrowSchemaPyCapsule(ArrowSchema* schema) { + return PyCapsule_New(schema, "arrow_schema", ReleaseArrowSchemaPyCapsule); + } + + .. tab-item:: Cython + + .. code-block:: cython + + import cpython + + cdef void release_arrow_schema_py_capsule(object schema_capsule): + cdef ArrowSchema* schema = <ArrowSchema*>cpython.PyCapsule_GetPointer( + schema_capsule, 'arrow_schema' + ) + if schema.release != NULL: + schema.release(schema) + + free(schema) + + cdef object make_arrow_schema_py_capsule(ArrowSchema* schema): + return cpython.PyCapsule_New( + <void*>schema, 'arrow_schema', release_arrow_schema_py_capsule + ) + + +Consume a PyCapsule +------------------- + +To consume a PyCapsule, use the `PyCapsule_GetPointer <https://docs.python.org/3/c-api/capsule.html#c.PyCapsule_GetPointer>`_ function +to get the pointer to the underlying struct. Import the struct using the your +systems Arrow C Data Interface import function. Only after that should the +capsule be freed. + +The below example shows how to consume a PyCapsule for an ``ArrowSchema``. The +code for ``ArrowArray`` and ``ArrowArrayStream`` is similar. + +.. tab-set:: + + .. tab-item:: C + + .. code-block:: c + + #include <Python.h> + + // If the capsule is not an ArrowSchema, will return NULL. + ArrowSchema* GetArrowSchemaPyCapsule(PyObject* capsule) { + return PyCapsule_GetPointer(schema, "arrow_schema"); + } + + .. tab-item:: Cython + + .. code-block:: cython + + import cpython + + cdef object get_arrow_schema_py_capsule(PyObject* capsule): + return cpython.PyCapsule_GetPointer(capsule, 'arrow_schema') + +Backwards Compatibility with PyArrow +------------------------------------ + +When interacting with PyArrow, the PyCapsule interface will be preferred over +the ``_export_to_c`` and ``_import_from_c`` methods. However, many libraries will +want to support a range of PyArrow versions. This can be done via Duck typing. + +For example, if your library had an import method such as: + +.. code-block:: python + + # OLD METHOD + def from_arrow(arr: pa.Array) + array_import_ptr = make_array_import_ptr() + schema_import_ptr = make_schema_import_ptr() + arr._export_to_c(array_import_ptr, schema_import_ptr) + return import_c_data(array_import_ptr, schema_import_ptr) + +You can rewrite this method to support both PyArrow and other libraries that +implement the PyCapsule interface: + +.. code-block:: python + + # NEW METHOD + def from_arrow(arr) + if hasattr(arr, "__arrow_c_array__"): + schema_ptr, array_ptr = arr.__arrow_c_array__() + return import_c_capsule_data(schema_ptr, array_ptr) + elif isinstance(arr, pa.Array): + # Deprecated method, used for older versions of PyArrow + array_import_ptr = make_array_import_ptr() + schema_import_ptr = make_schema_import_ptr() + arr._export_to_c(array_import_ptr, schema_import_ptr) + return import_c_data(array_import_ptr, schema_import_ptr) + else: + raise TypeError(f"Cannot import {type(arr)} as Arrow array data.") + +You may also wish to accept objects implementing the protocol in your +constructors. For example, in PyArrow, the :py:func:`array` and :py:func:`record_batch` +constructor accepts any object that implements the ``ArrowArrayExportable`` +protocol. Similarly, the PyArrow's :py:func:`schema` constructor accepts any object +that implements the ``ArrowSchemaExportable`` protocol. + +Now if your library has an export to PyArrow function, such as: + +.. code-block:: python + + # OLD METHOD + def to_arrow(self) -> pa.Array: + array_export_ptr = make_array_export_ptr() + schema_export_ptr = make_schema_export_ptr() + self.export_c_data(array_export_ptr, schema_export_ptr) + return pa.Array._import_from_c(array_export_ptr, schema_export_ptr) + +You can implement the protocol instead and encourage users to prefer directly +passing your object to the :py:func:`array` constructor, which accepts any object +that implements the protocol: + +.. code-block:: python + + import warnings + + # NEW METHOD + def to_arrow(self) -> pa.Array: + if hasattr(pa.Array, "__arrow_c_array__"): Review Comment: You might add a comment explaining why the `hasattr` call, because it was not immediately obvious for me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
