jorisvandenbossche commented on code in PR #37797: URL: https://github.com/apache/arrow/pull/37797#discussion_r1335990043
########## docs/source/format/CDataInterface/PyCapsuleInterface.rst: ########## @@ -0,0 +1,238 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +============================= +The Arrow PyCapsule Interface +============================= + +Rationale +========= + +The :ref:`C data interface <c-data-interface>` and +:ref:`C stream interface <c-stream-interface>` allow moving Arrow data between +different implementations of Arrow. However, these interfaces don't specify how +Python libraries should expose these structs to other libraries. Prior to this, +many libraries simply provided export to PyArrow data structures, using the +``_import_from_c`` and ``_export_from_c`` methods. However, this always required +PyArrow to be installed. + +This interface allows any library to export Arrow data structures to other +libraries that understand the same protocol. + +Goals +----- + +* Standardize the PyCapsule objects that represent ``ArrowSchema``, ``ArrowArray``, + and ``ArrowArrayStream``. +* Define standard methods to get these PyCapsules, so any Python objects that + support export via Arrow can be recognized by other libraries. + + +Non-goals +--------- + +* Standardizing what public APIs should be used for import. This is left up to + individual libraries. + + +Comparison to DataFrame Interchange Protocol +-------------------------------------------- + +:doc:`DataFrame Interchange Protocol <../dataframe.html>` is another protocol Review Comment: ```suggestion The :ref:`DataFrame Interchange Protocol <py-dataframe-interchange-protocol>` is another protocol ``` Not fully sure which page this is currently linking to? If you add `.. _py-dataframe-interchange-protocol:` to `interchange_protocol.rst`, the above siggestion should work ########## docs/source/format/CDataInterface/PyCapsuleInterface.rst: ########## @@ -0,0 +1,238 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +============================= +The Arrow PyCapsule Interface +============================= + +Rationale +========= + +The :ref:`C data interface <c-data-interface>` and +:ref:`C stream interface <c-stream-interface>` allow moving Arrow data between +different implementations of Arrow. However, these interfaces don't specify how +Python libraries should expose these structs to other libraries. Prior to this, +many libraries simply provided export to PyArrow data structures, using the +``_import_from_c`` and ``_export_from_c`` methods. However, this always required +PyArrow to be installed. + +This interface allows any library to export Arrow data structures to other +libraries that understand the same protocol. + +Goals +----- + +* Standardize the PyCapsule objects that represent ``ArrowSchema``, ``ArrowArray``, + and ``ArrowArrayStream``. +* Define standard methods to get these PyCapsules, so any Python objects that + support export via Arrow can be recognized by other libraries. + + +Non-goals +--------- + +* Standardizing what public APIs should be used for import. This is left up to + individual libraries. + + +Comparison to DataFrame Interchange Protocol +-------------------------------------------- + +:doc:`DataFrame Interchange Protocol <../dataframe.html>` is another protocol +in Python that allows for the sharing of data between libraries. This protocol +is complementary to the DataFrame Interchange Protocol. Many of the objects that +implement this protocol will also implement the DataFrame Interchange Protocol. + +This protocol is specific to Arrow-based data structures, while the DataFrame +Interchange Protocol allows non-Arrow data frames and arrays to be shared as well. +Because of this, these PyCapsules can support Arrow-specific features such as +nested columns. + +This protocol is also much more minimal than the DataFrame Interchange Protocol. +It just handles data export, rather than defining accessors for details like +number of rows or columns. + +In summary, if you are implementing this protocol, you should also consider +implementing the DataFrame Interchange Protocol. + + +PyCapsule Standard +================== + +`PyCapsules`_ allow for a ``name`` to be associated with the capsule, allowing +consumers to verify that the capsule contains the expected data. To make sure +Arrow structs are recognized, the following names should be used: + +.. list-table:: + :widths: 25 25 + :header-rows: 1 + + * - C Interface Type + - PyCapsule Name + * - ArrowSchema + - ``arrowschema`` + * - ArrowArray + - ``arrowarray`` + * - ArrowArrayStream + - ``arrowarraystream`` + + +Lifetime Semantics +------------------ + +The exported PyCapsules should have a destructor that calls the +:ref:`release callback <c-data-interface-released>` +of the Arrow struct, if it is not already null. This prevents a memory leak in +case the capsule was never passed to another consumer. + +If the consumer has been passed to a consumer, the consumer should have moved Review Comment: ```suggestion If the capsule has been passed to a consumer, the consumer should have moved ``` ########## docs/source/format/CDataInterface/PyCapsuleInterface.rst: ########## @@ -0,0 +1,238 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +============================= +The Arrow PyCapsule Interface +============================= + +Rationale +========= + +The :ref:`C data interface <c-data-interface>` and +:ref:`C stream interface <c-stream-interface>` allow moving Arrow data between +different implementations of Arrow. However, these interfaces don't specify how +Python libraries should expose these structs to other libraries. Prior to this, +many libraries simply provided export to PyArrow data structures, using the +``_import_from_c`` and ``_export_from_c`` methods. However, this always required +PyArrow to be installed. + +This interface allows any library to export Arrow data structures to other +libraries that understand the same protocol. + +Goals +----- + +* Standardize the PyCapsule objects that represent ``ArrowSchema``, ``ArrowArray``, + and ``ArrowArrayStream``. +* Define standard methods to get these PyCapsules, so any Python objects that + support export via Arrow can be recognized by other libraries. + + +Non-goals +--------- + +* Standardizing what public APIs should be used for import. This is left up to + individual libraries. + + +Comparison to DataFrame Interchange Protocol +-------------------------------------------- + +:doc:`DataFrame Interchange Protocol <../dataframe.html>` is another protocol +in Python that allows for the sharing of data between libraries. This protocol +is complementary to the DataFrame Interchange Protocol. Many of the objects that +implement this protocol will also implement the DataFrame Interchange Protocol. + +This protocol is specific to Arrow-based data structures, while the DataFrame +Interchange Protocol allows non-Arrow data frames and arrays to be shared as well. +Because of this, these PyCapsules can support Arrow-specific features such as +nested columns. + +This protocol is also much more minimal than the DataFrame Interchange Protocol. +It just handles data export, rather than defining accessors for details like +number of rows or columns. + +In summary, if you are implementing this protocol, you should also consider +implementing the DataFrame Interchange Protocol. + + +PyCapsule Standard +================== + +`PyCapsules`_ allow for a ``name`` to be associated with the capsule, allowing +consumers to verify that the capsule contains the expected data. To make sure +Arrow structs are recognized, the following names should be used: + +.. list-table:: + :widths: 25 25 + :header-rows: 1 + + * - C Interface Type + - PyCapsule Name + * - ArrowSchema + - ``arrowschema`` + * - ArrowArray + - ``arrowarray`` + * - ArrowArrayStream + - ``arrowarraystream`` + + +Lifetime Semantics +------------------ + +The exported PyCapsules should have a destructor that calls the +:ref:`release callback <c-data-interface-released>` +of the Arrow struct, if it is not already null. This prevents a memory leak in +case the capsule was never passed to another consumer. + +If the consumer has been passed to a consumer, the consumer should have moved +the data and marked the release callback as null, so there isn’t a risk of +releasing data the consumer is using. Read more in the C Data Interface +specification. + +Because of the lifetime semantics of the underlying Arrow structs, the +PyCapsules can only be used once. + + +Export Protocol +=============== + +The interface is three separate protocols: + +* ``ArrowSchemaExportable``, which defines the ``__arrow_c_schema__`` method. +* ``ArrowArrayExportable``, which defines the ``__arrow_c_array__``. It extends + ``ArrowSchemaExportable``, so it must also define ``__arrow_c_schema__``. +* ``ArrowStreamExportable``, which defines the ``__arrow_c_stream__``. It extends + ``ArrowSchemaExportable``, so it must also define ``__arrow_c_schema__``. + +The protocols are defined below in terms of ``typing.Protocol``. These may be +copied into a library for the purposes of static type checking, but this is not +required to implement the protocol. + +.. TODO: should stream have a batch size parameter? +.. TODO: should we have a allow_copy flag? + + + +.. code-block:: python + + from typing import Tuple, Protocol + from typing_extensions import Self + + class ArrowSchemaExportable(Protocol): + def __arrow_c_schema__(self) -> object: + """ + Get a PyCapsule containing a C ArrowSchema representation of the object. + + The capsule will have a name of "arrowschema". + """ + ... + + class ArrowArrayExportable(ArrowSchemaExportable, Protocol): + def __arrow_c_array__( + self, + requested_schema: object | None = None + ) -> Tuple[object, object]: + """ + Export array as a pair of PyCapsules for the ArrowSchema and ArrowArray. + + The capsule will have a name of "arrowarray". + + If requested_schema is passed, the callee should attempt to provide the + data in the requested schema. However, this is best-effort, and the + callee may return a PyCapsule containing an ArrowArray with a different + schema. This parameter is useful for cases where the underlying data + could be represented in multiple ways, and the caller has a preference + for how it is represented. For example, some systems have a single + integer type, but Arrow has multiple integer types with different + sizes and sign. Review Comment: Related to my comment on the design google doc, do we want to explicitly call out that the requested schema is expected to have the exact same structure (in terms of number of fields and field names) as the schema of the object, and that the callee is expected to error if that is not the case? ########## docs/source/format/CDataInterface/PyCapsuleInterface.rst: ########## @@ -0,0 +1,238 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +============================= +The Arrow PyCapsule Interface +============================= + +Rationale +========= + +The :ref:`C data interface <c-data-interface>` and +:ref:`C stream interface <c-stream-interface>` allow moving Arrow data between +different implementations of Arrow. However, these interfaces don't specify how +Python libraries should expose these structs to other libraries. Prior to this, +many libraries simply provided export to PyArrow data structures, using the +``_import_from_c`` and ``_export_from_c`` methods. However, this always required +PyArrow to be installed. + +This interface allows any library to export Arrow data structures to other +libraries that understand the same protocol. + +Goals +----- + +* Standardize the PyCapsule objects that represent ``ArrowSchema``, ``ArrowArray``, + and ``ArrowArrayStream``. +* Define standard methods to get these PyCapsules, so any Python objects that + support export via Arrow can be recognized by other libraries. + + +Non-goals +--------- + +* Standardizing what public APIs should be used for import. This is left up to + individual libraries. + + +Comparison to DataFrame Interchange Protocol +-------------------------------------------- + +:doc:`DataFrame Interchange Protocol <../dataframe.html>` is another protocol +in Python that allows for the sharing of data between libraries. This protocol +is complementary to the DataFrame Interchange Protocol. Many of the objects that +implement this protocol will also implement the DataFrame Interchange Protocol. + +This protocol is specific to Arrow-based data structures, while the DataFrame +Interchange Protocol allows non-Arrow data frames and arrays to be shared as well. +Because of this, these PyCapsules can support Arrow-specific features such as +nested columns. + +This protocol is also much more minimal than the DataFrame Interchange Protocol. +It just handles data export, rather than defining accessors for details like +number of rows or columns. + +In summary, if you are implementing this protocol, you should also consider +implementing the DataFrame Interchange Protocol. + + +PyCapsule Standard +================== + +`PyCapsules`_ allow for a ``name`` to be associated with the capsule, allowing +consumers to verify that the capsule contains the expected data. To make sure +Arrow structs are recognized, the following names should be used: + +.. list-table:: + :widths: 25 25 + :header-rows: 1 + + * - C Interface Type + - PyCapsule Name + * - ArrowSchema + - ``arrowschema`` + * - ArrowArray + - ``arrowarray`` + * - ArrowArrayStream + - ``arrowarraystream`` + + +Lifetime Semantics +------------------ + +The exported PyCapsules should have a destructor that calls the +:ref:`release callback <c-data-interface-released>` +of the Arrow struct, if it is not already null. This prevents a memory leak in +case the capsule was never passed to another consumer. + +If the consumer has been passed to a consumer, the consumer should have moved +the data and marked the release callback as null, so there isn’t a risk of +releasing data the consumer is using. Read more in the C Data Interface +specification. + +Because of the lifetime semantics of the underlying Arrow structs, the +PyCapsules can only be used once. + + +Export Protocol +=============== + +The interface is three separate protocols: + +* ``ArrowSchemaExportable``, which defines the ``__arrow_c_schema__`` method. +* ``ArrowArrayExportable``, which defines the ``__arrow_c_array__``. It extends + ``ArrowSchemaExportable``, so it must also define ``__arrow_c_schema__``. +* ``ArrowStreamExportable``, which defines the ``__arrow_c_stream__``. It extends + ``ArrowSchemaExportable``, so it must also define ``__arrow_c_schema__``. + +The protocols are defined below in terms of ``typing.Protocol``. These may be +copied into a library for the purposes of static type checking, but this is not +required to implement the protocol. + +.. TODO: should stream have a batch size parameter? +.. TODO: should we have a allow_copy flag? Review Comment: The batch size could potentially also be something that is handled in a protocol one level higher up? (like the Dataset protocol we were discussing, that sparked pushing this one forward) I imagine that this interface typically "just" exposes the data as they are, using the batch size as the data already exists in memory. Of course, use cases for an Arrow Stream typically involves data _not_ yet in memory, and there a batch size would be useful (but again, could also be handled by another protocol if you want to be able to control this) ########## docs/source/index.rst: ########## @@ -82,6 +82,7 @@ target environment.** format/CStreamInterface format/CDeviceDataInterface format/ADBC + format/Python Review Comment: You added it as a subdirectory "CDataInterface", so that should match with that here? ########## docs/source/format/CDataInterface/PyCapsuleInterface.rst: ########## @@ -0,0 +1,238 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +============================= +The Arrow PyCapsule Interface +============================= + +Rationale +========= + +The :ref:`C data interface <c-data-interface>` and +:ref:`C stream interface <c-stream-interface>` allow moving Arrow data between +different implementations of Arrow. However, these interfaces don't specify how +Python libraries should expose these structs to other libraries. Prior to this, +many libraries simply provided export to PyArrow data structures, using the +``_import_from_c`` and ``_export_from_c`` methods. However, this always required +PyArrow to be installed. + +This interface allows any library to export Arrow data structures to other +libraries that understand the same protocol. + +Goals +----- + +* Standardize the PyCapsule objects that represent ``ArrowSchema``, ``ArrowArray``, + and ``ArrowArrayStream``. +* Define standard methods to get these PyCapsules, so any Python objects that + support export via Arrow can be recognized by other libraries. + + +Non-goals +--------- + +* Standardizing what public APIs should be used for import. This is left up to + individual libraries. + + +Comparison to DataFrame Interchange Protocol +-------------------------------------------- + +:doc:`DataFrame Interchange Protocol <../dataframe.html>` is another protocol +in Python that allows for the sharing of data between libraries. This protocol +is complementary to the DataFrame Interchange Protocol. Many of the objects that +implement this protocol will also implement the DataFrame Interchange Protocol. + +This protocol is specific to Arrow-based data structures, while the DataFrame +Interchange Protocol allows non-Arrow data frames and arrays to be shared as well. +Because of this, these PyCapsules can support Arrow-specific features such as +nested columns. + +This protocol is also much more minimal than the DataFrame Interchange Protocol. +It just handles data export, rather than defining accessors for details like +number of rows or columns. + +In summary, if you are implementing this protocol, you should also consider +implementing the DataFrame Interchange Protocol. + + +PyCapsule Standard +================== + +`PyCapsules`_ allow for a ``name`` to be associated with the capsule, allowing +consumers to verify that the capsule contains the expected data. To make sure +Arrow structs are recognized, the following names should be used: + +.. list-table:: + :widths: 25 25 + :header-rows: 1 + + * - C Interface Type + - PyCapsule Name + * - ArrowSchema + - ``arrowschema`` + * - ArrowArray + - ``arrowarray`` + * - ArrowArrayStream + - ``arrowarraystream`` + + +Lifetime Semantics +------------------ + +The exported PyCapsules should have a destructor that calls the +:ref:`release callback <c-data-interface-released>` +of the Arrow struct, if it is not already null. This prevents a memory leak in +case the capsule was never passed to another consumer. + +If the consumer has been passed to a consumer, the consumer should have moved +the data and marked the release callback as null, so there isn’t a risk of +releasing data the consumer is using. Read more in the C Data Interface +specification. + +Because of the lifetime semantics of the underlying Arrow structs, the +PyCapsules can only be used once. + + +Export Protocol +=============== + +The interface is three separate protocols: + +* ``ArrowSchemaExportable``, which defines the ``__arrow_c_schema__`` method. +* ``ArrowArrayExportable``, which defines the ``__arrow_c_array__``. It extends + ``ArrowSchemaExportable``, so it must also define ``__arrow_c_schema__``. +* ``ArrowStreamExportable``, which defines the ``__arrow_c_stream__``. It extends + ``ArrowSchemaExportable``, so it must also define ``__arrow_c_schema__``. + +The protocols are defined below in terms of ``typing.Protocol``. These may be +copied into a library for the purposes of static type checking, but this is not +required to implement the protocol. + +.. TODO: should stream have a batch size parameter? +.. TODO: should we have a allow_copy flag? + + + +.. code-block:: python + + from typing import Tuple, Protocol + from typing_extensions import Self + + class ArrowSchemaExportable(Protocol): + def __arrow_c_schema__(self) -> object: + """ + Get a PyCapsule containing a C ArrowSchema representation of the object. + + The capsule will have a name of "arrowschema". + """ + ... + + class ArrowArrayExportable(ArrowSchemaExportable, Protocol): + def __arrow_c_array__( + self, + requested_schema: object | None = None + ) -> Tuple[object, object]: + """ + Export array as a pair of PyCapsules for the ArrowSchema and ArrowArray. + + The capsule will have a name of "arrowarray". + + If requested_schema is passed, the callee should attempt to provide the + data in the requested schema. However, this is best-effort, and the + callee may return a PyCapsule containing an ArrowArray with a different + schema. This parameter is useful for cases where the underlying data + could be represented in multiple ways, and the caller has a preference + for how it is represented. For example, some systems have a single + integer type, but Arrow has multiple integer types with different + sizes and sign. Review Comment: Ah, I see there is a section below expanding on this, thanks! ########## docs/source/format/CDataInterface/PyCapsuleInterface.rst: ########## @@ -0,0 +1,238 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +============================= +The Arrow PyCapsule Interface +============================= + +Rationale +========= + +The :ref:`C data interface <c-data-interface>` and +:ref:`C stream interface <c-stream-interface>` allow moving Arrow data between +different implementations of Arrow. However, these interfaces don't specify how +Python libraries should expose these structs to other libraries. Prior to this, +many libraries simply provided export to PyArrow data structures, using the +``_import_from_c`` and ``_export_from_c`` methods. However, this always required +PyArrow to be installed. + +This interface allows any library to export Arrow data structures to other +libraries that understand the same protocol. + +Goals +----- + +* Standardize the PyCapsule objects that represent ``ArrowSchema``, ``ArrowArray``, + and ``ArrowArrayStream``. +* Define standard methods to get these PyCapsules, so any Python objects that + support export via Arrow can be recognized by other libraries. + + +Non-goals +--------- + +* Standardizing what public APIs should be used for import. This is left up to + individual libraries. + + +Comparison to DataFrame Interchange Protocol +-------------------------------------------- + +:doc:`DataFrame Interchange Protocol <../dataframe.html>` is another protocol +in Python that allows for the sharing of data between libraries. This protocol +is complementary to the DataFrame Interchange Protocol. Many of the objects that +implement this protocol will also implement the DataFrame Interchange Protocol. + +This protocol is specific to Arrow-based data structures, while the DataFrame +Interchange Protocol allows non-Arrow data frames and arrays to be shared as well. +Because of this, these PyCapsules can support Arrow-specific features such as +nested columns. + +This protocol is also much more minimal than the DataFrame Interchange Protocol. +It just handles data export, rather than defining accessors for details like +number of rows or columns. + +In summary, if you are implementing this protocol, you should also consider +implementing the DataFrame Interchange Protocol. + + +PyCapsule Standard +================== + +`PyCapsules`_ allow for a ``name`` to be associated with the capsule, allowing +consumers to verify that the capsule contains the expected data. To make sure +Arrow structs are recognized, the following names should be used: + +.. list-table:: + :widths: 25 25 + :header-rows: 1 + + * - C Interface Type + - PyCapsule Name + * - ArrowSchema + - ``arrowschema`` + * - ArrowArray + - ``arrowarray`` + * - ArrowArrayStream + - ``arrowarraystream`` + + +Lifetime Semantics +------------------ + +The exported PyCapsules should have a destructor that calls the +:ref:`release callback <c-data-interface-released>` +of the Arrow struct, if it is not already null. This prevents a memory leak in +case the capsule was never passed to another consumer. + +If the consumer has been passed to a consumer, the consumer should have moved +the data and marked the release callback as null, so there isn’t a risk of +releasing data the consumer is using. Read more in the C Data Interface +specification. + +Because of the lifetime semantics of the underlying Arrow structs, the +PyCapsules can only be used once. + + +Export Protocol +=============== + +The interface is three separate protocols: + +* ``ArrowSchemaExportable``, which defines the ``__arrow_c_schema__`` method. +* ``ArrowArrayExportable``, which defines the ``__arrow_c_array__``. It extends + ``ArrowSchemaExportable``, so it must also define ``__arrow_c_schema__``. +* ``ArrowStreamExportable``, which defines the ``__arrow_c_stream__``. It extends + ``ArrowSchemaExportable``, so it must also define ``__arrow_c_schema__``. + +The protocols are defined below in terms of ``typing.Protocol``. These may be +copied into a library for the purposes of static type checking, but this is not +required to implement the protocol. + +.. TODO: should stream have a batch size parameter? +.. TODO: should we have a allow_copy flag? + + + +.. code-block:: python + + from typing import Tuple, Protocol + from typing_extensions import Self + + class ArrowSchemaExportable(Protocol): + def __arrow_c_schema__(self) -> object: + """ + Get a PyCapsule containing a C ArrowSchema representation of the object. + + The capsule will have a name of "arrowschema". + """ + ... + + class ArrowArrayExportable(ArrowSchemaExportable, Protocol): + def __arrow_c_array__( + self, + requested_schema: object | None = None + ) -> Tuple[object, object]: + """ + Export array as a pair of PyCapsules for the ArrowSchema and ArrowArray. + + The capsule will have a name of "arrowarray". Review Comment: ```suggestion The ArrowArray capsule will have a name of "arrowarray". ``` (to clarify given that the sentence above mentions two different capsules) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
