This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/arrow-nanoarrow.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8db33d2d update documentation for tag main
8db33d2d is described below
commit 8db33d2d583cf972c07f0f1841375e4c034f3e9d
Author: GitHub Actions <[email protected]>
AuthorDate: Fri May 17 19:24:05 2024 +0000
update documentation for tag main
---
main/_sources/getting-started/python.rst.txt | 319 ++++++++++++++++++++-------
main/getting-started/index.html | 1 +
main/getting-started/python.html | 283 +++++++++++++++++-------
main/r/pkgdown.yml | 2 +-
main/searchindex.js | 2 +-
5 files changed, 441 insertions(+), 166 deletions(-)
diff --git a/main/_sources/getting-started/python.rst.txt
b/main/_sources/getting-started/python.rst.txt
index c8bd5751..06babade 100644
--- a/main/_sources/getting-started/python.rst.txt
+++ b/main/_sources/getting-started/python.rst.txt
@@ -37,12 +37,21 @@ interfaces.
Installation
------------
-Python bindings for nanoarrow are not yet available on PyPI. You can
-install via URL (requires a C compiler):
+The nanoarrow Python bindings are available from
+`PyPI <https://pypi.org/>`__ and
+`conda-forge <https://conda-forge.org/>`__:
-.. code:: bash
+.. code:: shell
+
+ pip install nanoarrow
+ conda install nanoarrow -c conda-forge
+
+Development versions (based on the ``main`` branch) are also available:
+
+.. code:: shell
- python -m pip install
"git+https://github.com/apache/arrow-nanoarrow.git#egg=nanoarrow&subdirectory=python";
+ pip install --extra-index-url https://pypi.fury.io/arrow-nightlies/ \
+ --prefer-binary --pre nanoarrow
If you can import the namespace, you’re good to go!
@@ -50,14 +59,173 @@ If you can import the namespace, you’re good to go!
import nanoarrow as na
-Low-level C library bindings
-----------------------------
+Data types, arrays, and array streams
+-------------------------------------
The Arrow C Data and Arrow C Stream interfaces are comprised of three
structures: the ``ArrowSchema`` which represents a data type of an
array, the ``ArrowArray`` which represents the values of an array, and
an ``ArrowArrayStream``, which represents zero or more ``ArrowArray``\ s
-with a common ``ArrowSchema``.
+with a common ``ArrowSchema``. These concepts map to the
+``nanoarrow.Schema``, ``nanoarrow.Array``, and ``nanoarrow.ArrayStream``
+in the Python package.
+
+.. code:: python
+
+ na.int32()
+
+::
+
+ <Schema> int32
+
+.. code:: python
+
+ na.Array([1, 2, 3], na.int32())
+
+::
+
+ nanoarrow.Array<int32>[3]
+ 1
+ 2
+ 3
+
+The ``nanoarrow.Array`` can accommodate arrays with any number of
+chunks, reflecting the reality that many array containers (e.g.,
+``pyarrow.ChunkedArray``, ``polars.Series``) support this.
+
+.. code:: python
+
+ chunked = na.Array.from_chunks([[1, 2, 3], [4, 5, 6]], na.int32())
+ chunked
+
+::
+
+ nanoarrow.Array<int32>[6]
+ 1
+ 2
+ 3
+ 4
+ 5
+ 6
+
+Whereas chunks of an ``Array`` are always fully materialized when the
+object is constructed, the chunks of an ``ArrayStream`` have not
+necessarily been resolved yet.
+
+.. code:: python
+
+ stream = na.ArrayStream(chunked)
+ stream
+
+::
+
+ nanoarrow.ArrayStream<int32>
+
+.. code:: python
+
+ with stream:
+ for chunk in stream:
+ print(chunk)
+
+::
+
+ nanoarrow.Array<int32>[3]
+ 1
+ 2
+ 3
+ nanoarrow.Array<int32>[3]
+ 4
+ 5
+ 6
+
+The ``nanoarrow.ArrayStream`` also provides an interface to nanoarrow’s
+`Arrow
+IPC
<https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc>`__
+reader:
+
+.. code:: python
+
+ url =
"https://github.com/apache/arrow-experiments/raw/main/data/arrow-commits/arrow-commits.arrows";
+ na.ArrayStream.from_url(url)
+
+::
+
+ nanoarrow.ArrayStream<non-nullable struct<commit: string, time:
timestamp('us', 'UTC'), files: int3...>
+
+These objects implement the `Arrow PyCapsule
+interface
<https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html>`__
+for both producing and consuming and are interchangeable with
+``pyarrow`` objects in many cases:
+
+.. code:: python
+
+ import pyarrow as pa
+
+ pa.field(na.int32())
+
+::
+
+ pyarrow.Field<: int32>
+
+.. code:: python
+
+ pa.chunked_array(chunked)
+
+::
+
+ <pyarrow.lib.ChunkedArray object at 0x12a49a250>
+ [
+ [
+ 1,
+ 2,
+ 3
+ ],
+ [
+ 4,
+ 5,
+ 6
+ ]
+ ]
+
+.. code:: python
+
+ pa.array(chunked.chunk(1))
+
+::
+
+ <pyarrow.lib.Int32Array object at 0x11b552500>
+ [
+ 4,
+ 5,
+ 6
+ ]
+
+.. code:: python
+
+ na.Array(pa.array([10, 11, 12]))
+
+::
+
+ nanoarrow.Array<int64>[3]
+ 10
+ 11
+ 12
+
+.. code:: python
+
+ na.Schema(pa.string())
+
+::
+
+ <Schema> string
+
+Low-level C library bindings
+----------------------------
+
+The nanoarrow Python package also provides lower level wrappers around
+Arrow C interface structures. You can create these using
+``nanoarrow.c_schema()``, ``nanoarrow.c_array()``, and
+``nanoarrow.c_array_stream()``.
Schemas
~~~~~~~
@@ -70,13 +238,11 @@ Interface
<https://arrow.apache.org/docs/format/CDataInterface.html>`__
.. code:: python
- import pyarrow as pa
- schema = na.c_schema(pa.decimal128(10, 3))
- schema
+ na.c_schema(pa.decimal128(10, 3))
::
- <nanoarrow.c_lib.CSchema decimal128(10, 3)>
+ <nanoarrow.c_schema.CSchema decimal128(10, 3)>
- format: 'd:10,3'
- name: ''
- flags: 2
@@ -84,45 +250,18 @@ Interface
<https://arrow.apache.org/docs/format/CDataInterface.html>`__
- dictionary: NULL
- children[0]:
-You can extract the fields of a ``CSchema`` object one at a time or
-parse it into a view to extract deserialized parameters.
+Using ``c_schema()`` is a good fit for testing and for ephemeral schema
+objects that are being passed from one library to another. To extract
+the fields of a schema in a more convenient form, use ``Schema()``:
.. code:: python
- na.c_schema_view(schema)
+ schema = na.Schema(pa.decimal128(10, 3))
+ schema.precision, schema.scale
::
- <nanoarrow.c_lib.CSchemaView>
- - type: 'decimal128'
- - storage_type: 'decimal128'
- - decimal_bitwidth: 128
- - decimal_precision: 10
- - decimal_scale: 3
- - dictionary_ordered: False
- - map_keys_sorted: False
- - nullable: True
- - storage_type_id: 24
- - type_id: 24
-
-Advanced users can allocate an empty ``CSchema`` and populate its
-contents by passing its ``._addr()`` to a schema-exporting function.
-
-.. code:: python
-
- schema = na.allocate_c_schema()
- pa.int32()._export_to_c(schema._addr())
- schema
-
-::
-
- <nanoarrow.c_lib.CSchema int32>
- - format: 'i'
- - name: ''
- - flags: 2
- - metadata: NULL
- - dictionary: NULL
- - children[0]:
+ (10, 3)
The ``CSchema`` object cleans up after itself: when the object is
deleted, the underlying ``ArrowSchema`` is released.
@@ -139,52 +278,61 @@ Interface
<https://arrow.apache.org/docs/format/CDataInterface.html>`__
.. code:: python
- array = na.c_array(pa.array(["one", "two", "three", None]))
- array
+ na.c_array(["one", "two", "three", None], na.string())
::
- <nanoarrow.c_lib.CArray string>
+ <nanoarrow.c_array.CArray string>
- length: 4
- offset: 0
- null_count: 1
- - buffers: (3678035706048, 3678035705984, 3678035706112)
+ - buffers: (4754305168, 4754307808, 4754310464)
- dictionary: NULL
- children[0]:
-You can extract the fields of a ``CArray`` one at a time or parse it
-into a view to extract deserialized content:
+Using ``c_array()`` is a good fit for testing and for ephemeral array
+objects that are being passed from one library to another. For a higher
+level interface, use ``Array()``:
.. code:: python
- na.c_array_view(array)
+ array = na.Array(["one", "two", "three", None], na.string())
+ array.to_pylist()
::
- <nanoarrow.c_lib.CArrayView>
- - storage_type: 'string'
- - length: 4
- - offset: 0
- - null_count: 1
- - buffers[3]:
- - validity <bool[1 b] 11100000>
- - data_offset <int32[20 b] 0 3 6 11 11>
- - data <string[11 b] b'onetwothree'>
- - dictionary: NULL
- - children[0]:
+ ['one', 'two', 'three', None]
+
+.. code:: python
+
+ array.buffers
+
+::
+
+ (nanoarrow.c_lib.CBufferView(bool[1 b] 11100000),
+ nanoarrow.c_lib.CBufferView(int32[20 b] 0 3 6 11 11),
+ nanoarrow.c_lib.CBufferView(string[11 b] b'onetwothree'))
-Like the ``CSchema``, you can allocate an empty one and access its
-address with ``_addr()`` to pass to other array-exporting functions.
+Advanced users can create arrays directly from buffers using
+``c_array_from_buffers()``:
.. code:: python
- array = na.allocate_c_array()
- pa.array([1, 2, 3])._export_to_c(array._addr(), array.schema._addr())
- array.length
+ na.c_array_from_buffers(
+ na.string(),
+ 2,
+ [None, na.c_buffer([0, 3, 6], na.int32()), b"abcdef"]
+ )
::
- 3
+ <nanoarrow.c_array.CArray string>
+ - length: 2
+ - offset: 0
+ - null_count: 0
+ - buffers: (0, 5002908320, 4999694624)
+ - dictionary: NULL
+ - children[0]:
Array streams
~~~~~~~~~~~~~
@@ -194,20 +342,19 @@ representing a sequence of ``CArray``\ s with a common
``CSchema`` to an
``ArrowArrayStream`` and wrap it as a Python object. This works for any
object implementing the `Arrow PyCapsule
Interface <https://arrow.apache.org/docs/format/CDataInterface.html>`__
-(e.g., ``pyarrow.RecordBatchReader``).
+(e.g., ``pyarrow.RecordBatchReader``, ``pyarrow.ChunkedArray``).
.. code:: python
- pa_array_child = pa.array([1, 2, 3], pa.int32())
- pa_array = pa.record_batch([pa_array_child], names=["some_column"])
- reader = pa.RecordBatchReader.from_batches(pa_array.schema, [pa_array])
+ pa_batch = pa.record_batch({"col1": [1, 2, 3]})
+ reader = pa.RecordBatchReader.from_batches(pa_batch.schema, [pa_batch])
array_stream = na.c_array_stream(reader)
array_stream
::
- <nanoarrow.c_lib.CArrayStream>
- - get_schema(): struct<some_column: int32>
+ <nanoarrow.c_array_stream.CArrayStream>
+ - get_schema(): struct<col1: int64>
You can pull the next array from the stream using ``.get_next()`` or use
it like an iterator. The ``.get_next()`` method will raise
@@ -220,34 +367,34 @@ it like an iterator. The ``.get_next()`` method will raise
::
- <nanoarrow.c_lib.CArray struct<some_column: int32>>
+ <nanoarrow.c_array.CArray struct<col1: int64>>
- length: 3
- offset: 0
- null_count: 0
- buffers: (0,)
- dictionary: NULL
- children[1]:
- 'some_column': <nanoarrow.c_lib.CArray int32>
+ 'col1': <nanoarrow.c_array.CArray int64>
- length: 3
- offset: 0
- null_count: 0
- - buffers: (0, 3678035837056)
+ - buffers: (0, 2642948588352)
- dictionary: NULL
- children[0]:
-You can also get the address of a freshly-allocated stream to pass to a
-suitable exporting function:
+Use ``ArrayStream()`` for a higher level interface:
.. code:: python
- array_stream = na.allocate_c_array_stream()
- reader._export_to_c(array_stream._addr())
- array_stream
+ reader = pa.RecordBatchReader.from_batches(pa_batch.schema, [pa_batch])
+ na.ArrayStream(reader).read_all()
::
- <nanoarrow.c_lib.CArrayStream>
- - get_schema(): struct<some_column: int32>
+ nanoarrow.Array<non-nullable struct<col1: int64>>[3]
+ {'col1': 1}
+ {'col1': 2}
+ {'col1': 3}
Development
-----------
@@ -271,3 +418,7 @@ Tests use `pytest <https://docs.pytest.org/>`__:
# Run tests
pytest -vvx
+
+CMake is currently required to ensure that the vendored copy of
+nanoarrow in the Python package stays in sync with the nanoarrow sources
+in the working tree.
diff --git a/main/getting-started/index.html b/main/getting-started/index.html
index b9217838..ff60c3f4 100644
--- a/main/getting-started/index.html
+++ b/main/getting-started/index.html
@@ -404,6 +404,7 @@ document.write(`
</li>
<li class="toctree-l1"><a class="reference internal"
href="python.html">Python</a><ul>
<li class="toctree-l2"><a class="reference internal"
href="python.html#installation">Installation</a></li>
+<li class="toctree-l2"><a class="reference internal"
href="python.html#data-types-arrays-and-array-streams">Data types, arrays, and
array streams</a></li>
<li class="toctree-l2"><a class="reference internal"
href="python.html#low-level-c-library-bindings">Low-level C library
bindings</a></li>
<li class="toctree-l2"><a class="reference internal"
href="python.html#development">Development</a></li>
</ul>
diff --git a/main/getting-started/python.html b/main/getting-started/python.html
index e760ba15..a711c9c4 100644
--- a/main/getting-started/python.html
+++ b/main/getting-started/python.html
@@ -417,9 +417,16 @@ Stream</a>
interfaces.</p>
<section id="installation">
<h2>Installation<a class="headerlink" href="#installation" title="Link to this
heading">#</a></h2>
-<p>Python bindings for nanoarrow are not yet available on PyPI. You can
-install via URL (requires a C compiler):</p>
-<div class="highlight-bash notranslate"><div
class="highlight"><pre><span></span>python<span class="w"> </span>-m<span
class="w"> </span>pip<span class="w"> </span>install<span class="w">
</span><span
class="s2">"git+https://github.com/apache/arrow-nanoarrow.git#egg=nanoarrow&subdirectory=python"</span>
+<p>The nanoarrow Python bindings are available from
+<a class="reference external" href="https://pypi.org/";>PyPI</a> and
+<a class="reference external"
href="https://conda-forge.org/";>conda-forge</a>:</p>
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>pip<span class="w"> </span>install<span
class="w"> </span>nanoarrow
+conda<span class="w"> </span>install<span class="w"> </span>nanoarrow<span
class="w"> </span>-c<span class="w"> </span>conda-forge
+</pre></div>
+</div>
+<p>Development versions (based on the <code class="docutils literal
notranslate"><span class="pre">main</span></code> branch) are also
available:</p>
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>pip<span class="w"> </span>install<span
class="w"> </span>--extra-index-url<span class="w">
</span>https://pypi.fury.io/arrow-nightlies/<span class="w"> </span><span
class="se">\</span>
+<span class="w"> </span>--prefer-binary<span class="w"> </span>--pre<span
class="w"> </span>nanoarrow
</pre></div>
</div>
<p>If you can import the namespace, you’re good to go!</p>
@@ -427,13 +434,145 @@ install via URL (requires a C compiler):</p>
</pre></div>
</div>
</section>
-<section id="low-level-c-library-bindings">
-<h2>Low-level C library bindings<a class="headerlink"
href="#low-level-c-library-bindings" title="Link to this heading">#</a></h2>
+<section id="data-types-arrays-and-array-streams">
+<h2>Data types, arrays, and array streams<a class="headerlink"
href="#data-types-arrays-and-array-streams" title="Link to this
heading">#</a></h2>
<p>The Arrow C Data and Arrow C Stream interfaces are comprised of three
structures: the <code class="docutils literal notranslate"><span
class="pre">ArrowSchema</span></code> which represents a data type of an
array, the <code class="docutils literal notranslate"><span
class="pre">ArrowArray</span></code> which represents the values of an array,
and
an <code class="docutils literal notranslate"><span
class="pre">ArrowArrayStream</span></code>, which represents zero or more <code
class="docutils literal notranslate"><span class="pre">ArrowArray</span></code>s
-with a common <code class="docutils literal notranslate"><span
class="pre">ArrowSchema</span></code>.</p>
+with a common <code class="docutils literal notranslate"><span
class="pre">ArrowSchema</span></code>. These concepts map to the
+<code class="docutils literal notranslate"><span
class="pre">nanoarrow.Schema</span></code>, <code class="docutils literal
notranslate"><span class="pre">nanoarrow.Array</span></code>, and <code
class="docutils literal notranslate"><span
class="pre">nanoarrow.ArrayStream</span></code>
+in the Python package.</p>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">na</span><span
class="o">.</span><span class="n">int32</span><span class="p">()</span>
+</pre></div>
+</div>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">Schema</span><span class="o">></span> <span class="n">int32</span>
+</pre></div>
+</div>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">na</span><span
class="o">.</span><span class="n">Array</span><span class="p">([</span><span
class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span
class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span
class="n">na</span><span class="o">.</span><span class="n">int32</span><span
class="p">())</span>
+</pre></div>
+</div>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">nanoarrow</span><span
class="o">.</span><span class="n">Array</span><span class="o"><</span><span
class="n">int32</span><span class="o">></span><span class="p">[</span><span
class="mi">3</span><span class="p">]</span>
+<span class="mi">1</span>
+<span class="mi">2</span>
+<span class="mi">3</span>
+</pre></div>
+</div>
+<p>The <code class="docutils literal notranslate"><span
class="pre">nanoarrow.Array</span></code> can accommodate arrays with any
number of
+chunks, reflecting the reality that many array containers (e.g.,
+<code class="docutils literal notranslate"><span
class="pre">pyarrow.ChunkedArray</span></code>, <code class="docutils literal
notranslate"><span class="pre">polars.Series</span></code>) support this.</p>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">chunked</span> <span
class="o">=</span> <span class="n">na</span><span class="o">.</span><span
class="n">Array</span><span class="o">.</span><span
class="n">from_chunks</span><span class="p">([[</span><span
class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span
class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span
class="p">[</span><span class="mi"> [...]
+<span class="n">chunked</span>
+</pre></div>
+</div>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">nanoarrow</span><span
class="o">.</span><span class="n">Array</span><span class="o"><</span><span
class="n">int32</span><span class="o">></span><span class="p">[</span><span
class="mi">6</span><span class="p">]</span>
+<span class="mi">1</span>
+<span class="mi">2</span>
+<span class="mi">3</span>
+<span class="mi">4</span>
+<span class="mi">5</span>
+<span class="mi">6</span>
+</pre></div>
+</div>
+<p>Whereas chunks of an <code class="docutils literal notranslate"><span
class="pre">Array</span></code> are always fully materialized when the
+object is constructed, the chunks of an <code class="docutils literal
notranslate"><span class="pre">ArrayStream</span></code> have not
+necessarily been resolved yet.</p>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">stream</span> <span
class="o">=</span> <span class="n">na</span><span class="o">.</span><span
class="n">ArrayStream</span><span class="p">(</span><span
class="n">chunked</span><span class="p">)</span>
+<span class="n">stream</span>
+</pre></div>
+</div>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">nanoarrow</span><span
class="o">.</span><span class="n">ArrayStream</span><span
class="o"><</span><span class="n">int32</span><span class="o">></span>
+</pre></div>
+</div>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="k">with</span> <span
class="n">stream</span><span class="p">:</span>
+ <span class="k">for</span> <span class="n">chunk</span> <span
class="ow">in</span> <span class="n">stream</span><span class="p">:</span>
+ <span class="nb">print</span><span class="p">(</span><span
class="n">chunk</span><span class="p">)</span>
+</pre></div>
+</div>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">nanoarrow</span><span
class="o">.</span><span class="n">Array</span><span class="o"><</span><span
class="n">int32</span><span class="o">></span><span class="p">[</span><span
class="mi">3</span><span class="p">]</span>
+<span class="mi">1</span>
+<span class="mi">2</span>
+<span class="mi">3</span>
+<span class="n">nanoarrow</span><span class="o">.</span><span
class="n">Array</span><span class="o"><</span><span
class="n">int32</span><span class="o">></span><span class="p">[</span><span
class="mi">3</span><span class="p">]</span>
+<span class="mi">4</span>
+<span class="mi">5</span>
+<span class="mi">6</span>
+</pre></div>
+</div>
+<p>The <code class="docutils literal notranslate"><span
class="pre">nanoarrow.ArrayStream</span></code> also provides an interface to
nanoarrow’s
+<a class="reference external"
href="https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc";>Arrow
+IPC</a>
+reader:</p>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">url</span> <span
class="o">=</span> <span
class="s2">"https://github.com/apache/arrow-experiments/raw/main/data/arrow-commits/arrow-commits.arrows"</span>
+<span class="n">na</span><span class="o">.</span><span
class="n">ArrayStream</span><span class="o">.</span><span
class="n">from_url</span><span class="p">(</span><span
class="n">url</span><span class="p">)</span>
+</pre></div>
+</div>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">nanoarrow</span><span
class="o">.</span><span class="n">ArrayStream</span><span
class="o"><</span><span class="n">non</span><span class="o">-</span><span
class="n">nullable</span> <span class="n">struct</span><span
class="o"><</span><span class="n">commit</span><span class="p">:</span>
<span class="n">string</span><span class="p">,</span> <span
class="n">time</span><span class="p">:< [...]
+</pre></div>
+</div>
+<p>These objects implement the <a class="reference external"
href="https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html";>Arrow
PyCapsule
+interface</a>
+for both producing and consuming and are interchangeable with
+<code class="docutils literal notranslate"><span
class="pre">pyarrow</span></code> objects in many cases:</p>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="kn">import</span> <span
class="nn">pyarrow</span> <span class="k">as</span> <span class="nn">pa</span>
+
+<span class="n">pa</span><span class="o">.</span><span
class="n">field</span><span class="p">(</span><span class="n">na</span><span
class="o">.</span><span class="n">int32</span><span class="p">())</span>
+</pre></div>
+</div>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">pyarrow</span><span
class="o">.</span><span class="n">Field</span><span class="o"><</span><span
class="p">:</span> <span class="n">int32</span><span class="o">></span>
+</pre></div>
+</div>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">pa</span><span
class="o">.</span><span class="n">chunked_array</span><span
class="p">(</span><span class="n">chunked</span><span class="p">)</span>
+</pre></div>
+</div>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">pyarrow</span><span class="o">.</span><span class="n">lib</span><span
class="o">.</span><span class="n">ChunkedArray</span> <span
class="nb">object</span> <span class="n">at</span> <span
class="mh">0x12a49a250</span><span class="o">></span>
+<span class="p">[</span>
+ <span class="p">[</span>
+ <span class="mi">1</span><span class="p">,</span>
+ <span class="mi">2</span><span class="p">,</span>
+ <span class="mi">3</span>
+ <span class="p">],</span>
+ <span class="p">[</span>
+ <span class="mi">4</span><span class="p">,</span>
+ <span class="mi">5</span><span class="p">,</span>
+ <span class="mi">6</span>
+ <span class="p">]</span>
+<span class="p">]</span>
+</pre></div>
+</div>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">pa</span><span
class="o">.</span><span class="n">array</span><span class="p">(</span><span
class="n">chunked</span><span class="o">.</span><span
class="n">chunk</span><span class="p">(</span><span class="mi">1</span><span
class="p">))</span>
+</pre></div>
+</div>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">pyarrow</span><span class="o">.</span><span class="n">lib</span><span
class="o">.</span><span class="n">Int32Array</span> <span
class="nb">object</span> <span class="n">at</span> <span
class="mh">0x11b552500</span><span class="o">></span>
+<span class="p">[</span>
+ <span class="mi">4</span><span class="p">,</span>
+ <span class="mi">5</span><span class="p">,</span>
+ <span class="mi">6</span>
+<span class="p">]</span>
+</pre></div>
+</div>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">na</span><span
class="o">.</span><span class="n">Array</span><span class="p">(</span><span
class="n">pa</span><span class="o">.</span><span class="n">array</span><span
class="p">([</span><span class="mi">10</span><span class="p">,</span> <span
class="mi">11</span><span class="p">,</span> <span class="mi">12</span><span
class="p">]))</span>
+</pre></div>
+</div>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">nanoarrow</span><span
class="o">.</span><span class="n">Array</span><span class="o"><</span><span
class="n">int64</span><span class="o">></span><span class="p">[</span><span
class="mi">3</span><span class="p">]</span>
+<span class="mi">10</span>
+<span class="mi">11</span>
+<span class="mi">12</span>
+</pre></div>
+</div>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">na</span><span
class="o">.</span><span class="n">Schema</span><span class="p">(</span><span
class="n">pa</span><span class="o">.</span><span class="n">string</span><span
class="p">())</span>
+</pre></div>
+</div>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">Schema</span><span class="o">></span> <span class="n">string</span>
+</pre></div>
+</div>
+</section>
+<section id="low-level-c-library-bindings">
+<h2>Low-level C library bindings<a class="headerlink"
href="#low-level-c-library-bindings" title="Link to this heading">#</a></h2>
+<p>The nanoarrow Python package also provides lower level wrappers around
+Arrow C interface structures. You can create these using
+<code class="docutils literal notranslate"><span
class="pre">nanoarrow.c_schema()</span></code>, <code class="docutils literal
notranslate"><span class="pre">nanoarrow.c_array()</span></code>, and
+<code class="docutils literal notranslate"><span
class="pre">nanoarrow.c_array_stream()</span></code>.</p>
<section id="schemas">
<h3>Schemas<a class="headerlink" href="#schemas" title="Link to this
heading">#</a></h3>
<p>Use <code class="docutils literal notranslate"><span
class="pre">nanoarrow.c_schema()</span></code> to convert an object to an <code
class="docutils literal notranslate"><span class="pre">ArrowSchema</span></code>
@@ -441,12 +580,10 @@ and wrap it as a Python object. This works for any object
implementing
the <a class="reference external"
href="https://arrow.apache.org/docs/format/CDataInterface.html";>Arrow PyCapsule
Interface</a>
(e.g., <code class="docutils literal notranslate"><span
class="pre">pyarrow.Schema</span></code>, <code class="docutils literal
notranslate"><span class="pre">pyarrow.DataType</span></code>, and <code
class="docutils literal notranslate"><span
class="pre">pyarrow.Field</span></code>).</p>
-<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="kn">import</span> <span
class="nn">pyarrow</span> <span class="k">as</span> <span class="nn">pa</span>
-<span class="n">schema</span> <span class="o">=</span> <span
class="n">na</span><span class="o">.</span><span class="n">c_schema</span><span
class="p">(</span><span class="n">pa</span><span class="o">.</span><span
class="n">decimal128</span><span class="p">(</span><span
class="mi">10</span><span class="p">,</span> <span class="mi">3</span><span
class="p">))</span>
-<span class="n">schema</span>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">na</span><span
class="o">.</span><span class="n">c_schema</span><span class="p">(</span><span
class="n">pa</span><span class="o">.</span><span
class="n">decimal128</span><span class="p">(</span><span
class="mi">10</span><span class="p">,</span> <span class="mi">3</span><span
class="p">))</span>
</pre></div>
</div>
-<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_lib</span><span class="o">.</span><span class="n">CSchema</span>
<span class="n">decimal128</span><span class="p">(</span><span
class="mi">10</span><span class="p">,</span> <span class="mi">3</span><span
class="p">)</span><span class="o">></span>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_schema</span><span class="o">.</span><span class="n">CSchema</span>
<span class="n">decimal128</span><span class="p">(</span><span
class="mi">10</span><span class="p">,</span> <span class="mi">3</span><span
class="p">)</span><span class="o">></span>
<span class="o">-</span> <span class="nb">format</span><span
class="p">:</span> <span class="s1">'d:10,3'</span>
<span class="o">-</span> <span class="n">name</span><span class="p">:</span>
<span class="s1">''</span>
<span class="o">-</span> <span class="n">flags</span><span class="p">:</span>
<span class="mi">2</span>
@@ -455,38 +592,14 @@ Interface</a>
<span class="o">-</span> <span class="n">children</span><span
class="p">[</span><span class="mi">0</span><span class="p">]:</span>
</pre></div>
</div>
-<p>You can extract the fields of a <code class="docutils literal
notranslate"><span class="pre">CSchema</span></code> object one at a time or
-parse it into a view to extract deserialized parameters.</p>
-<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">na</span><span
class="o">.</span><span class="n">c_schema_view</span><span
class="p">(</span><span class="n">schema</span><span class="p">)</span>
-</pre></div>
-</div>
-<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_lib</span><span class="o">.</span><span
class="n">CSchemaView</span><span class="o">></span>
-<span class="o">-</span> <span class="nb">type</span><span class="p">:</span>
<span class="s1">'decimal128'</span>
-<span class="o">-</span> <span class="n">storage_type</span><span
class="p">:</span> <span class="s1">'decimal128'</span>
-<span class="o">-</span> <span class="n">decimal_bitwidth</span><span
class="p">:</span> <span class="mi">128</span>
-<span class="o">-</span> <span class="n">decimal_precision</span><span
class="p">:</span> <span class="mi">10</span>
-<span class="o">-</span> <span class="n">decimal_scale</span><span
class="p">:</span> <span class="mi">3</span>
-<span class="o">-</span> <span class="n">dictionary_ordered</span><span
class="p">:</span> <span class="kc">False</span>
-<span class="o">-</span> <span class="n">map_keys_sorted</span><span
class="p">:</span> <span class="kc">False</span>
-<span class="o">-</span> <span class="n">nullable</span><span
class="p">:</span> <span class="kc">True</span>
-<span class="o">-</span> <span class="n">storage_type_id</span><span
class="p">:</span> <span class="mi">24</span>
-<span class="o">-</span> <span class="n">type_id</span><span
class="p">:</span> <span class="mi">24</span>
-</pre></div>
-</div>
-<p>Advanced users can allocate an empty <code class="docutils literal
notranslate"><span class="pre">CSchema</span></code> and populate its
-contents by passing its <code class="docutils literal notranslate"><span
class="pre">._addr()</span></code> to a schema-exporting function.</p>
-<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">schema</span> <span
class="o">=</span> <span class="n">na</span><span class="o">.</span><span
class="n">allocate_c_schema</span><span class="p">()</span>
-<span class="n">pa</span><span class="o">.</span><span
class="n">int32</span><span class="p">()</span><span class="o">.</span><span
class="n">_export_to_c</span><span class="p">(</span><span
class="n">schema</span><span class="o">.</span><span
class="n">_addr</span><span class="p">())</span>
-<span class="n">schema</span>
+<p>Using <code class="docutils literal notranslate"><span
class="pre">c_schema()</span></code> is a good fit for testing and for
ephemeral schema
+objects that are being passed from one library to another. To extract
+the fields of a schema in a more convenient form, use <code class="docutils
literal notranslate"><span class="pre">Schema()</span></code>:</p>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">schema</span> <span
class="o">=</span> <span class="n">na</span><span class="o">.</span><span
class="n">Schema</span><span class="p">(</span><span class="n">pa</span><span
class="o">.</span><span class="n">decimal128</span><span
class="p">(</span><span class="mi">10</span><span class="p">,</span> <span
class="mi">3</span><span class="p">))</span>
+<span class="n">schema</span><span class="o">.</span><span
class="n">precision</span><span class="p">,</span> <span
class="n">schema</span><span class="o">.</span><span class="n">scale</span>
</pre></div>
</div>
-<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_lib</span><span class="o">.</span><span class="n">CSchema</span>
<span class="n">int32</span><span class="o">></span>
-<span class="o">-</span> <span class="nb">format</span><span
class="p">:</span> <span class="s1">'i'</span>
-<span class="o">-</span> <span class="n">name</span><span class="p">:</span>
<span class="s1">''</span>
-<span class="o">-</span> <span class="n">flags</span><span class="p">:</span>
<span class="mi">2</span>
-<span class="o">-</span> <span class="n">metadata</span><span
class="p">:</span> <span class="n">NULL</span>
-<span class="o">-</span> <span class="n">dictionary</span><span
class="p">:</span> <span class="n">NULL</span>
-<span class="o">-</span> <span class="n">children</span><span
class="p">[</span><span class="mi">0</span><span class="p">]:</span>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="p">(</span><span
class="mi">10</span><span class="p">,</span> <span class="mi">3</span><span
class="p">)</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span
class="pre">CSchema</span></code> object cleans up after itself: when the
object is
@@ -500,45 +613,52 @@ can be used to interpret its contents. This works for any
object
implementing the <a class="reference external"
href="https://arrow.apache.org/docs/format/CDataInterface.html";>Arrow PyCapsule
Interface</a>
(e.g., <code class="docutils literal notranslate"><span
class="pre">pyarrow.Array</span></code>, <code class="docutils literal
notranslate"><span class="pre">pyarrow.RecordBatch</span></code>).</p>
-<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">array</span> <span
class="o">=</span> <span class="n">na</span><span class="o">.</span><span
class="n">c_array</span><span class="p">(</span><span class="n">pa</span><span
class="o">.</span><span class="n">array</span><span class="p">([</span><span
class="s2">"one"</span><span class="p">,</span> <span
class="s2">"two"</span><span class="p">,</span> <span
class="s2">" [...]
-<span class="n">array</span>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">na</span><span
class="o">.</span><span class="n">c_array</span><span class="p">([</span><span
class="s2">"one"</span><span class="p">,</span> <span
class="s2">"two"</span><span class="p">,</span> <span
class="s2">"three"</span><span class="p">,</span> <span
class="kc">None</span><span class="p">],</span> <span class="n">na</span><span
class="o">.</span><span cl [...]
</pre></div>
</div>
-<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_lib</span><span class="o">.</span><span class="n">CArray</span>
<span class="n">string</span><span class="o">></span>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_array</span><span class="o">.</span><span class="n">CArray</span>
<span class="n">string</span><span class="o">></span>
<span class="o">-</span> <span class="n">length</span><span class="p">:</span>
<span class="mi">4</span>
<span class="o">-</span> <span class="n">offset</span><span class="p">:</span>
<span class="mi">0</span>
<span class="o">-</span> <span class="n">null_count</span><span
class="p">:</span> <span class="mi">1</span>
-<span class="o">-</span> <span class="n">buffers</span><span
class="p">:</span> <span class="p">(</span><span
class="mi">3678035706048</span><span class="p">,</span> <span
class="mi">3678035705984</span><span class="p">,</span> <span
class="mi">3678035706112</span><span class="p">)</span>
+<span class="o">-</span> <span class="n">buffers</span><span
class="p">:</span> <span class="p">(</span><span
class="mi">4754305168</span><span class="p">,</span> <span
class="mi">4754307808</span><span class="p">,</span> <span
class="mi">4754310464</span><span class="p">)</span>
<span class="o">-</span> <span class="n">dictionary</span><span
class="p">:</span> <span class="n">NULL</span>
<span class="o">-</span> <span class="n">children</span><span
class="p">[</span><span class="mi">0</span><span class="p">]:</span>
</pre></div>
</div>
-<p>You can extract the fields of a <code class="docutils literal
notranslate"><span class="pre">CArray</span></code> one at a time or parse it
-into a view to extract deserialized content:</p>
-<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">na</span><span
class="o">.</span><span class="n">c_array_view</span><span
class="p">(</span><span class="n">array</span><span class="p">)</span>
+<p>Using <code class="docutils literal notranslate"><span
class="pre">c_array()</span></code> is a good fit for testing and for ephemeral
array
+objects that are being passed from one library to another. For a higher
+level interface, use <code class="docutils literal notranslate"><span
class="pre">Array()</span></code>:</p>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">array</span> <span
class="o">=</span> <span class="n">na</span><span class="o">.</span><span
class="n">Array</span><span class="p">([</span><span
class="s2">"one"</span><span class="p">,</span> <span
class="s2">"two"</span><span class="p">,</span> <span
class="s2">"three"</span><span class="p">,</span> <span
class="kc">None</span><span class="p">],</span> <span [...]
+<span class="n">array</span><span class="o">.</span><span
class="n">to_pylist</span><span class="p">()</span>
</pre></div>
</div>
-<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_lib</span><span class="o">.</span><span
class="n">CArrayView</span><span class="o">></span>
-<span class="o">-</span> <span class="n">storage_type</span><span
class="p">:</span> <span class="s1">'string'</span>
-<span class="o">-</span> <span class="n">length</span><span class="p">:</span>
<span class="mi">4</span>
-<span class="o">-</span> <span class="n">offset</span><span class="p">:</span>
<span class="mi">0</span>
-<span class="o">-</span> <span class="n">null_count</span><span
class="p">:</span> <span class="mi">1</span>
-<span class="o">-</span> <span class="n">buffers</span><span
class="p">[</span><span class="mi">3</span><span class="p">]:</span>
- <span class="o">-</span> <span class="n">validity</span> <span
class="o"><</span><span class="nb">bool</span><span class="p">[</span><span
class="mi">1</span> <span class="n">b</span><span class="p">]</span> <span
class="mi">11100000</span><span class="o">></span>
- <span class="o">-</span> <span class="n">data_offset</span> <span
class="o"><</span><span class="n">int32</span><span class="p">[</span><span
class="mi">20</span> <span class="n">b</span><span class="p">]</span> <span
class="mi">0</span> <span class="mi">3</span> <span class="mi">6</span> <span
class="mi">11</span> <span class="mi">11</span><span class="o">></span>
- <span class="o">-</span> <span class="n">data</span> <span
class="o"><</span><span class="n">string</span><span class="p">[</span><span
class="mi">11</span> <span class="n">b</span><span class="p">]</span> <span
class="sa">b</span><span class="s1">'onetwothree'</span><span
class="o">></span>
-<span class="o">-</span> <span class="n">dictionary</span><span
class="p">:</span> <span class="n">NULL</span>
-<span class="o">-</span> <span class="n">children</span><span
class="p">[</span><span class="mi">0</span><span class="p">]:</span>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="p">[</span><span
class="s1">'one'</span><span class="p">,</span> <span
class="s1">'two'</span><span class="p">,</span> <span
class="s1">'three'</span><span class="p">,</span> <span
class="kc">None</span><span class="p">]</span>
</pre></div>
</div>
-<p>Like the <code class="docutils literal notranslate"><span
class="pre">CSchema</span></code>, you can allocate an empty one and access its
-address with <code class="docutils literal notranslate"><span
class="pre">_addr()</span></code> to pass to other array-exporting
functions.</p>
-<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">array</span> <span
class="o">=</span> <span class="n">na</span><span class="o">.</span><span
class="n">allocate_c_array</span><span class="p">()</span>
-<span class="n">pa</span><span class="o">.</span><span
class="n">array</span><span class="p">([</span><span class="mi">1</span><span
class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span
class="mi">3</span><span class="p">])</span><span class="o">.</span><span
class="n">_export_to_c</span><span class="p">(</span><span
class="n">array</span><span class="o">.</span><span class="n">_addr</span><span
class="p">(),</span> <span class="n">array</span><span class="o">.</spa [...]
-<span class="n">array</span><span class="o">.</span><span
class="n">length</span>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">array</span><span
class="o">.</span><span class="n">buffers</span>
</pre></div>
</div>
-<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="mi">3</span>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="p">(</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_lib</span><span class="o">.</span><span
class="n">CBufferView</span><span class="p">(</span><span
class="nb">bool</span><span class="p">[</span><span class="mi">1</span> <span
class="n">b</span><span class="p">]</span> <span
class="mi">11100000</span><span class="p">),</span>
+ <span class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_lib</span><span class="o">.</span><span
class="n">CBufferView</span><span class="p">(</span><span
class="n">int32</span><span class="p">[</span><span class="mi">20</span> <span
class="n">b</span><span class="p">]</span> <span class="mi">0</span> <span
class="mi">3</span> <span class="mi">6</span> <span class="mi">11</span> <span
class="mi">11</span><span class="p">),</span>
+ <span class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_lib</span><span class="o">.</span><span
class="n">CBufferView</span><span class="p">(</span><span
class="n">string</span><span class="p">[</span><span class="mi">11</span> <span
class="n">b</span><span class="p">]</span> <span class="sa">b</span><span
class="s1">'onetwothree'</span><span class="p">))</span>
+</pre></div>
+</div>
+<p>Advanced users can create arrays directly from buffers using
+<code class="docutils literal notranslate"><span
class="pre">c_array_from_buffers()</span></code>:</p>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">na</span><span
class="o">.</span><span class="n">c_array_from_buffers</span><span
class="p">(</span>
+ <span class="n">na</span><span class="o">.</span><span
class="n">string</span><span class="p">(),</span>
+ <span class="mi">2</span><span class="p">,</span>
+ <span class="p">[</span><span class="kc">None</span><span
class="p">,</span> <span class="n">na</span><span class="o">.</span><span
class="n">c_buffer</span><span class="p">([</span><span
class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span
class="p">,</span> <span class="mi">6</span><span class="p">],</span> <span
class="n">na</span><span class="o">.</span><span class="n">int32</span><span
class="p">()),</span> <span class="sa">b</span><span class="s2">"ab [...]
+<span class="p">)</span>
+</pre></div>
+</div>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_array</span><span class="o">.</span><span class="n">CArray</span>
<span class="n">string</span><span class="o">></span>
+<span class="o">-</span> <span class="n">length</span><span class="p">:</span>
<span class="mi">2</span>
+<span class="o">-</span> <span class="n">offset</span><span class="p">:</span>
<span class="mi">0</span>
+<span class="o">-</span> <span class="n">null_count</span><span
class="p">:</span> <span class="mi">0</span>
+<span class="o">-</span> <span class="n">buffers</span><span
class="p">:</span> <span class="p">(</span><span class="mi">0</span><span
class="p">,</span> <span class="mi">5002908320</span><span class="p">,</span>
<span class="mi">4999694624</span><span class="p">)</span>
+<span class="o">-</span> <span class="n">dictionary</span><span
class="p">:</span> <span class="n">NULL</span>
+<span class="o">-</span> <span class="n">children</span><span
class="p">[</span><span class="mi">0</span><span class="p">]:</span>
</pre></div>
</div>
</section>
@@ -549,16 +669,15 @@ representing a sequence of <code class="docutils literal
notranslate"><span clas
<code class="docutils literal notranslate"><span
class="pre">ArrowArrayStream</span></code> and wrap it as a Python object. This
works for any
object implementing the <a class="reference external"
href="https://arrow.apache.org/docs/format/CDataInterface.html";>Arrow PyCapsule
Interface</a>
-(e.g., <code class="docutils literal notranslate"><span
class="pre">pyarrow.RecordBatchReader</span></code>).</p>
-<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">pa_array_child</span> <span
class="o">=</span> <span class="n">pa</span><span class="o">.</span><span
class="n">array</span><span class="p">([</span><span class="mi">1</span><span
class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span
class="mi">3</span><span class="p">],</span> <span class="n">pa</span><span
class="o">.</span><span class="n">int32</span><span class="p"> [...]
-<span class="n">pa_array</span> <span class="o">=</span> <span
class="n">pa</span><span class="o">.</span><span
class="n">record_batch</span><span class="p">([</span><span
class="n">pa_array_child</span><span class="p">],</span> <span
class="n">names</span><span class="o">=</span><span class="p">[</span><span
class="s2">"some_column"</span><span class="p">])</span>
-<span class="n">reader</span> <span class="o">=</span> <span
class="n">pa</span><span class="o">.</span><span
class="n">RecordBatchReader</span><span class="o">.</span><span
class="n">from_batches</span><span class="p">(</span><span
class="n">pa_array</span><span class="o">.</span><span
class="n">schema</span><span class="p">,</span> <span class="p">[</span><span
class="n">pa_array</span><span class="p">])</span>
+(e.g., <code class="docutils literal notranslate"><span
class="pre">pyarrow.RecordBatchReader</span></code>, <code class="docutils
literal notranslate"><span class="pre">pyarrow.ChunkedArray</span></code>).</p>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">pa_batch</span> <span
class="o">=</span> <span class="n">pa</span><span class="o">.</span><span
class="n">record_batch</span><span class="p">({</span><span
class="s2">"col1"</span><span class="p">:</span> <span
class="p">[</span><span class="mi">1</span><span class="p">,</span> <span
class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span
class="p">]})</span>
+<span class="n">reader</span> <span class="o">=</span> <span
class="n">pa</span><span class="o">.</span><span
class="n">RecordBatchReader</span><span class="o">.</span><span
class="n">from_batches</span><span class="p">(</span><span
class="n">pa_batch</span><span class="o">.</span><span
class="n">schema</span><span class="p">,</span> <span class="p">[</span><span
class="n">pa_batch</span><span class="p">])</span>
<span class="n">array_stream</span> <span class="o">=</span> <span
class="n">na</span><span class="o">.</span><span
class="n">c_array_stream</span><span class="p">(</span><span
class="n">reader</span><span class="p">)</span>
<span class="n">array_stream</span>
</pre></div>
</div>
-<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_lib</span><span class="o">.</span><span
class="n">CArrayStream</span><span class="o">></span>
-<span class="o">-</span> <span class="n">get_schema</span><span
class="p">():</span> <span class="n">struct</span><span
class="o"><</span><span class="n">some_column</span><span class="p">:</span>
<span class="n">int32</span><span class="o">></span>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_array_stream</span><span class="o">.</span><span
class="n">CArrayStream</span><span class="o">></span>
+<span class="o">-</span> <span class="n">get_schema</span><span
class="p">():</span> <span class="n">struct</span><span
class="o"><</span><span class="n">col1</span><span class="p">:</span> <span
class="n">int64</span><span class="o">></span>
</pre></div>
</div>
<p>You can pull the next array from the stream using <code class="docutils
literal notranslate"><span class="pre">.get_next()</span></code> or use
@@ -568,31 +687,31 @@ it like an iterator. The <code class="docutils literal
notranslate"><span class=
<span class="nb">print</span><span class="p">(</span><span
class="n">array</span><span class="p">)</span>
</pre></div>
</div>
-<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_lib</span><span class="o">.</span><span class="n">CArray</span>
<span class="n">struct</span><span class="o"><</span><span
class="n">some_column</span><span class="p">:</span> <span
class="n">int32</span><span class="o">>></span>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_array</span><span class="o">.</span><span class="n">CArray</span>
<span class="n">struct</span><span class="o"><</span><span
class="n">col1</span><span class="p">:</span> <span class="n">int64</span><span
class="o">>></span>
<span class="o">-</span> <span class="n">length</span><span class="p">:</span>
<span class="mi">3</span>
<span class="o">-</span> <span class="n">offset</span><span class="p">:</span>
<span class="mi">0</span>
<span class="o">-</span> <span class="n">null_count</span><span
class="p">:</span> <span class="mi">0</span>
<span class="o">-</span> <span class="n">buffers</span><span
class="p">:</span> <span class="p">(</span><span class="mi">0</span><span
class="p">,)</span>
<span class="o">-</span> <span class="n">dictionary</span><span
class="p">:</span> <span class="n">NULL</span>
<span class="o">-</span> <span class="n">children</span><span
class="p">[</span><span class="mi">1</span><span class="p">]:</span>
- <span class="s1">'some_column'</span><span class="p">:</span> <span
class="o"><</span><span class="n">nanoarrow</span><span
class="o">.</span><span class="n">c_lib</span><span class="o">.</span><span
class="n">CArray</span> <span class="n">int32</span><span class="o">></span>
+ <span class="s1">'col1'</span><span class="p">:</span> <span
class="o"><</span><span class="n">nanoarrow</span><span
class="o">.</span><span class="n">c_array</span><span class="o">.</span><span
class="n">CArray</span> <span class="n">int64</span><span class="o">></span>
<span class="o">-</span> <span class="n">length</span><span
class="p">:</span> <span class="mi">3</span>
<span class="o">-</span> <span class="n">offset</span><span
class="p">:</span> <span class="mi">0</span>
<span class="o">-</span> <span class="n">null_count</span><span
class="p">:</span> <span class="mi">0</span>
- <span class="o">-</span> <span class="n">buffers</span><span
class="p">:</span> <span class="p">(</span><span class="mi">0</span><span
class="p">,</span> <span class="mi">3678035837056</span><span class="p">)</span>
+ <span class="o">-</span> <span class="n">buffers</span><span
class="p">:</span> <span class="p">(</span><span class="mi">0</span><span
class="p">,</span> <span class="mi">2642948588352</span><span class="p">)</span>
<span class="o">-</span> <span class="n">dictionary</span><span
class="p">:</span> <span class="n">NULL</span>
<span class="o">-</span> <span class="n">children</span><span
class="p">[</span><span class="mi">0</span><span class="p">]:</span>
</pre></div>
</div>
-<p>You can also get the address of a freshly-allocated stream to pass to a
-suitable exporting function:</p>
-<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">array_stream</span> <span
class="o">=</span> <span class="n">na</span><span class="o">.</span><span
class="n">allocate_c_array_stream</span><span class="p">()</span>
-<span class="n">reader</span><span class="o">.</span><span
class="n">_export_to_c</span><span class="p">(</span><span
class="n">array_stream</span><span class="o">.</span><span
class="n">_addr</span><span class="p">())</span>
-<span class="n">array_stream</span>
+<p>Use <code class="docutils literal notranslate"><span
class="pre">ArrayStream()</span></code> for a higher level interface:</p>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="n">reader</span> <span
class="o">=</span> <span class="n">pa</span><span class="o">.</span><span
class="n">RecordBatchReader</span><span class="o">.</span><span
class="n">from_batches</span><span class="p">(</span><span
class="n">pa_batch</span><span class="o">.</span><span
class="n">schema</span><span class="p">,</span> <span class="p">[</span><span
class="n">pa_batch</span><span class="p">])</span>
+<span class="n">na</span><span class="o">.</span><span
class="n">ArrayStream</span><span class="p">(</span><span
class="n">reader</span><span class="p">)</span><span class="o">.</span><span
class="n">read_all</span><span class="p">()</span>
</pre></div>
</div>
-<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o"><</span><span
class="n">nanoarrow</span><span class="o">.</span><span
class="n">c_lib</span><span class="o">.</span><span
class="n">CArrayStream</span><span class="o">></span>
-<span class="o">-</span> <span class="n">get_schema</span><span
class="p">():</span> <span class="n">struct</span><span
class="o"><</span><span class="n">some_column</span><span class="p">:</span>
<span class="n">int32</span><span class="o">></span>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">nanoarrow</span><span
class="o">.</span><span class="n">Array</span><span class="o"><</span><span
class="n">non</span><span class="o">-</span><span class="n">nullable</span>
<span class="n">struct</span><span class="o"><</span><span
class="n">col1</span><span class="p">:</span> <span class="n">int64</span><span
class="o">>></span><span class="p">[</span><span class="mi">3</span [...]
+<span class="p">{</span><span class="s1">'col1'</span><span
class="p">:</span> <span class="mi">1</span><span class="p">}</span>
+<span class="p">{</span><span class="s1">'col1'</span><span
class="p">:</span> <span class="mi">2</span><span class="p">}</span>
+<span class="p">{</span><span class="s1">'col1'</span><span
class="p">:</span> <span class="mi">3</span><span class="p">}</span>
</pre></div>
</div>
</section>
@@ -615,6 +734,9 @@ pip<span class="w"> </span>install<span class="w">
</span>-e<span class="w"> </s
pytest<span class="w"> </span>-vvx
</pre></div>
</div>
+<p>CMake is currently required to ensure that the vendored copy of
+nanoarrow in the Python package stays in sync with the nanoarrow sources
+in the working tree.</p>
</section>
</section>
@@ -665,6 +787,7 @@ pytest<span class="w"> </span>-vvx
<nav class="bd-toc-nav page-toc"
aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#installation">Installation</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#data-types-arrays-and-array-streams">Data types, arrays, and array
streams</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#low-level-c-library-bindings">Low-level C library bindings</a><ul
class="visible nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#schemas">Schemas</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#arrays">Arrays</a></li>
diff --git a/main/r/pkgdown.yml b/main/r/pkgdown.yml
index 74fa9864..6fd6d709 100644
--- a/main/r/pkgdown.yml
+++ b/main/r/pkgdown.yml
@@ -2,5 +2,5 @@ pandoc: 3.1.3
pkgdown: 2.0.9
pkgdown_sha: ~
articles: {}
-last_built: 2024-05-17T19:12Z
+last_built: 2024-05-17T19:23Z
diff --git a/main/searchindex.js b/main/searchindex.js
index edd3037a..f455a4ec 100644
--- a/main/searchindex.js
+++ b/main/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"API Reference": [[8, "api-reference"]], "Array
Stream utilities": [[6, "array-stream-utilities"]], "Array Streams": [[3,
"array-streams"]], "Array streams": [[2, "array-streams"]], "Arrays": [[2,
"arrays"], [3, "arrays"]], "Arrow C Data Interface": [[5,
"arrow-c-data-interface"]], "Arrow C Device Interface": [[7,
"arrow-c-device-interface"]], "Arrow C data/nanoarrow interface basics": [[0,
"arrow-c-data-nanoarrow-interface-basics"]], "Bitmap utilities": [[ [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"API Reference": [[8, "api-reference"]], "Array
Stream utilities": [[6, "array-stream-utilities"]], "Array Streams": [[3,
"array-streams"]], "Array streams": [[2, "array-streams"]], "Arrays": [[2,
"arrays"], [3, "arrays"]], "Arrow C Data Interface": [[5,
"arrow-c-data-interface"]], "Arrow C Device Interface": [[7,
"arrow-c-device-interface"]], "Arrow C data/nanoarrow interface basics": [[0,
"arrow-c-data-nanoarrow-interface-basics"]], "Bitmap utilities": [[ [...]
\ No newline at end of file
