This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-python.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new 9894b5e0 Publish built docs triggered by
e6f6e66c1d180246ad933f8bcc0d40faa8426dfa
9894b5e0 is described below
commit 9894b5e0b6eb3d219decb1bfad5577041e198296
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Feb 21 21:08:50 2025 +0000
Publish built docs triggered by e6f6e66c1d180246ad933f8bcc0d40faa8426dfa
---
_sources/contributor-guide/ffi.rst.txt | 212 +++++++
_sources/index.rst.txt | 1 +
autoapi/datafusion/catalog/index.html | 5 +
autoapi/datafusion/context/index.html | 5 +
autoapi/datafusion/dataframe/index.html | 5 +
autoapi/datafusion/expr/index.html | 5 +
autoapi/datafusion/functions/index.html | 5 +
autoapi/datafusion/index.html | 5 +
autoapi/datafusion/input/base/index.html | 5 +
autoapi/datafusion/input/index.html | 5 +
autoapi/datafusion/input/location/index.html | 5 +
autoapi/datafusion/object_store/index.html | 5 +
autoapi/datafusion/plan/index.html | 5 +
autoapi/datafusion/record_batch/index.html | 5 +
autoapi/datafusion/substrait/index.html | 5 +
autoapi/datafusion/udf/index.html | 5 +
autoapi/index.html | 11 +-
contributor-guide/ffi.html | 639 +++++++++++++++++++++
contributor-guide/introduction.html | 11 +-
genindex.html | 5 +
index.html | 5 +
objects.inv | Bin 5945 -> 5964 bytes
py-modindex.html | 5 +
search.html | 5 +
searchindex.js | 2 +-
user-guide/basics.html | 5 +
user-guide/common-operations/aggregations.html | 67 ++-
user-guide/common-operations/basic-info.html | 51 +-
user-guide/common-operations/expressions.html | 5 +
user-guide/common-operations/functions.html | 25 +-
user-guide/common-operations/index.html | 5 +
user-guide/common-operations/joins.html | 5 +
.../common-operations/select-and-filter.html | 5 +
user-guide/common-operations/udf-and-udfa.html | 5 +
user-guide/common-operations/windows.html | 33 +-
user-guide/configuration.html | 5 +
user-guide/data-sources.html | 5 +
user-guide/introduction.html | 5 +
user-guide/io/arrow.html | 5 +
user-guide/io/avro.html | 5 +
user-guide/io/csv.html | 5 +
user-guide/io/index.html | 5 +
user-guide/io/json.html | 5 +
user-guide/io/parquet.html | 5 +
user-guide/io/table_provider.html | 5 +
user-guide/sql.html | 5 +
46 files changed, 1142 insertions(+), 85 deletions(-)
diff --git a/_sources/contributor-guide/ffi.rst.txt
b/_sources/contributor-guide/ffi.rst.txt
new file mode 100644
index 00000000..c1f9806b
--- /dev/null
+++ b/_sources/contributor-guide/ffi.rst.txt
@@ -0,0 +1,212 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+Python Extensions
+=================
+
+The DataFusion in Python project is designed to allow users to extend its
functionality in a few core
+areas. Ideally many users would like to package their extensions as a Python
package and easily
+integrate that package with this project. This page serves to describe some of
the challenges we face
+when doing these integrations and the approach our project uses.
+
+The Primary Issue
+-----------------
+
+Suppose you wish to use DataFusion and you have a custom data source that can
produce tables that
+can then be queried against, similar to how you can register a :ref:`CSV
<io_csv>` or
+:ref:`Parquet <io_parquet>` file. In DataFusion terminology, you likely want
to implement a
+:ref:`Custom Table Provider <io_custom_table_provider>`. In an effort to make
your data source
+as performant as possible and to utilize the features of DataFusion, you may
decide to write
+your source in Rust and then expose it through `PyO3 <https://pyo3.rs>`_ as a
Python library.
+
+At first glance, it may appear the best way to do this is to add the
``datafusion-python``
+crate as a dependency, provide a ``PyTable``, and then to register it with the
+``SessionContext``. Unfortunately, this will not work.
+
+When you produce your code as a Python library and it needs to interact with
the DataFusion
+library, at the lowest level they communicate through an Application Binary
Interface (ABI).
+The acronym sounds similar to API (Application Programming Interface), but it
is distinctly
+different.
+
+The ABI sets the standard for how these libraries can share data and functions
between each
+other. One of the key differences between Rust and other programming languages
is that Rust
+does not have a stable ABI. What this means in practice is that if you compile
a Rust library
+with one version of the ``rustc`` compiler and I compile another library to
interface with it
+but I use a different version of the compiler, there is no guarantee the
interface will be
+the same.
+
+In practice, this means that a Python library built with ``datafusion-python``
as a Rust
+dependency will generally **not** be compatible with the DataFusion Python
package, even
+if they reference the same version of ``datafusion-python``. If you attempt to
do this, it may
+work on your local computer if you have built both packages with the same
optimizations.
+This can sometimes lead to a false expectation that the code will work, but it
frequently
+breaks the moment you try to use your package against the released packages.
+
+You can find more information about the Rust ABI in their
+`online documentation <https://doc.rust-lang.org/reference/abi.html>`_.
+
+The FFI Approach
+----------------
+
+Rust supports interacting with other programming languages through it's
Foreign Function
+Interface (FFI). The advantage of using the FFI is that it enables you to
write data structures
+and functions that have a stable ABI. The allows you to use Rust code with C,
Python, and
+other languages. In fact, the `PyO3 <https://pyo3.rs>`_ library uses the FFI
to share data
+and functions between Python and Rust.
+
+The approach we are taking in the DataFusion in Python project is to
incrementally expose
+more portions of the DataFusion project via FFI interfaces. This allows users
to write Rust
+code that does **not** require the ``datafusion-python`` crate as a
dependency, expose their
+code in Python via PyO3, and have it interact with the DataFusion Python
package.
+
+Early adopters of this approach include `delta-rs
<https://delta-io.github.io/delta-rs/>`_
+who has adapted their Table Provider for use in ```datafusion-python``` with
only a few lines
+of code. Also, the DataFusion Python project uses the existing definitions from
+`Apache Arrow CStream Interface
<https://arrow.apache.org/docs/format/CStreamInterface.html>`_
+to support importing **and** exporting tables. Any Python package that
supports reading
+the Arrow C Stream interface can work with DataFusion Python out of the box!
You can read
+more about working with Arrow sources in the :ref:`Data Sources
<user_guide_data_sources>`
+page.
+
+To learn more about the Foreign Function Interface in Rust, the
+`Rustonomicon <https://doc.rust-lang.org/nomicon/ffi.html>`_ is a good
resource.
+
+Inspiration from Arrow
+----------------------
+
+DataFusion is built upon `Apache Arrow <https://arrow.apache.org/>`_. The
canonical Python
+Arrow implementation, `pyarrow
<https://arrow.apache.org/docs/python/index.html>`_ provides
+an excellent way to share Arrow data between Python projects without
performing any copy
+operations on the data. They do this by using a well defined set of
interfaces. You can
+find the details about their stream interface
+`here <https://arrow.apache.org/docs/format/CStreamInterface.html>`_. The
+`Rust Arrow Implementation <https://github.com/apache/arrow-rs>`_ also
supports these
+``C`` style definitions via the Foreign Function Interface.
+
+In addition to using these interfaces to transfer Arrow data between
libraries, ``pyarrow``
+goes one step further to make sharing the interfaces easier in Python. They do
this
+by exposing PyCapsules that contain the expected functionality.
+
+You can learn more about PyCapsules from the official
+`Python online documentation <https://docs.python.org/3/c-api/capsule.html>`_.
PyCapsules
+have excellent support in PyO3 already. The
+`PyO3 online documentation
<https://pyo3.rs/main/doc/pyo3/types/struct.pycapsule>`_ is a good source
+for more details on using PyCapsules in Rust.
+
+Two lessons we leverage from the Arrow project in DataFusion Python are:
+
+- We reuse the existing Arrow FFI functionality wherever possible.
+- We expose PyCapsules that contain a FFI stable struct.
+
+Implementation Details
+----------------------
+
+The bulk of the code necessary to perform our FFI operations is in the
upstream
+`DataFusion <https://datafusion.apache.org/>`_ core repository. You can review
the code and
+documentation in the `datafusion-ffi`_ crate.
+
+Our FFI implementation is narrowly focused at sharing data and functions with
Rust backed
+libraries. This allows us to use the `abi_stable crate
<https://crates.io/crates/abi_stable>`_.
+This is an excellent crate that allows for easy conversion between Rust native
types
+and FFI-safe alternatives. For example, if you needed to pass a
``Vec<String>`` via FFI,
+you can simply convert it to a ``RVec<RString>`` in an intuitive manner. It
also supports
+features like ``RResult`` and ``ROption`` that do not have an obvious
translation to a
+C equivalent.
+
+The `datafusion-ffi`_ crate has been designed to make it easy to convert from
DataFusion
+traits into their FFI counterparts. For example, if you have defined a custom
+`TableProvider
<https://docs.rs/datafusion/45.0.0/datafusion/catalog/trait.TableProvider.html>`_
+and you want to create a sharable FFI counterpart, you could write:
+
+.. code-block:: rust
+
+ let my_provider = MyTableProvider::default();
+ let ffi_provider = FFI_TableProvider::new(Arc::new(my_provider), false,
None);
+
+If you were interfacing with a library that provided the above
``FFI_TableProvider`` and
+you needed to turn it back into an ``TableProvider``, you can turn it into a
+``ForeignTableProvider`` with implements the ``TableProvider`` trait.
+
+.. code-block:: rust
+
+ let foreign_provider: ForeignTableProvider = ffi_provider.into();
+
+If you review the code in `datafusion-ffi`_ you will find that each of the
traits we share
+across the boundary has two portions, one with a ``FFI_`` prefix and one with
a ``Foreign``
+prefix. This is used to distinguish which side of the FFI boundary that struct
is
+designed to be used on. The structures with the ``FFI_`` prefix are to be used
on the
+**provider** of the structure. In the example we're showing, this means the
code that has
+written the underlying ``TableProvider`` implementation to access your custom
data source.
+The structures with the ``Foreign`` prefix are to be used by the receiver. In
this case,
+it is the ``datafusion-python`` library.
+
+In order to share these FFI structures, we need to wrap them in some kind of
Python object
+that can be used to interface from one package to another. As described in the
above
+section on our inspiration from Arrow, we use ``PyCapsule``. We can create a
``PyCapsule``
+for our provider thusly:
+
+.. code-block:: rust
+
+ let name = CString::new("datafusion_table_provider")?;
+ let my_capsule = PyCapsule::new_bound(py, provider, Some(name))?;
+
+On the receiving side, turn this pycapsule object into the
``FFI_TableProvider``, which
+can then be turned into a ``ForeignTableProvider`` the associated code is:
+
+.. code-block:: rust
+
+ let capsule = capsule.downcast::<PyCapsule>()?;
+ let provider = unsafe { capsule.reference::<FFI_TableProvider>() };
+
+By convention the ``datafusion-python`` library expects a Python object that
has a
+``TableProvider`` PyCapsule to have this capsule accessible by calling a
function named
+``__datafusion_table_provider__``. You can see a complete working example of
how to
+share a ``TableProvider`` from one python library to DataFusion Python in the
+`repository examples folder
<https://github.com/apache/datafusion-python/tree/main/examples/ffi-table-provider>`_.
+
+This section has been written using ``TableProvider`` as an example. It is the
first
+extension that has been written using this approach and the most thoroughly
implemented.
+As we continue to expose more of the DataFusion features, we intend to follow
this same
+design pattern.
+
+Alternative Approach
+--------------------
+
+Suppose you needed to expose some other features of DataFusion and you could
not wait
+for the upstream repository to implement the FFI approach we describe. In this
case
+you decide to create your dependency on the ``datafusion-python`` crate
instead.
+
+As we discussed, this is not guaranteed to work across different compiler
versions and
+optimization levels. If you wish to go down this route, there are two
approaches we
+have identified you can use.
+
+#. Re-export all of ``datafusion-python`` yourself with your extensions built
in.
+#. Carefully synchonize your software releases with the ``datafusion-python``
CI build
+ system so that your libraries use the exact same compiler, features, and
+ optimization level.
+
+We currently do not recommend either of these approaches as they are difficult
to
+maintain over a long period. Additionally, they require a tight version
coupling
+between libraries.
+
+Status of Work
+--------------
+
+At the time of this writing, the FFI features are under active development. To
see
+the latest status, we recommend reviewing the code in the `datafusion-ffi`_
crate.
+
+.. _datafusion-ffi: https://crates.io/crates/datafusion-ffi
diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt
index 34eb23b2..558b2d57 100644
--- a/_sources/index.rst.txt
+++ b/_sources/index.rst.txt
@@ -85,6 +85,7 @@ Example
:caption: CONTRIBUTOR GUIDE
contributor-guide/introduction
+ contributor-guide/ffi
.. _toc.api:
.. toctree::
diff --git a/autoapi/datafusion/catalog/index.html
b/autoapi/datafusion/catalog/index.html
index 3f250b3d..36c952b1 100644
--- a/autoapi/datafusion/catalog/index.html
+++ b/autoapi/datafusion/catalog/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/context/index.html
b/autoapi/datafusion/context/index.html
index e2026edb..e2490f47 100644
--- a/autoapi/datafusion/context/index.html
+++ b/autoapi/datafusion/context/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/dataframe/index.html
b/autoapi/datafusion/dataframe/index.html
index da5003d9..8faad512 100644
--- a/autoapi/datafusion/dataframe/index.html
+++ b/autoapi/datafusion/dataframe/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/expr/index.html
b/autoapi/datafusion/expr/index.html
index 78ecdc6c..93510fcb 100644
--- a/autoapi/datafusion/expr/index.html
+++ b/autoapi/datafusion/expr/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/functions/index.html
b/autoapi/datafusion/functions/index.html
index 6ddd0a95..9d0bf9c0 100644
--- a/autoapi/datafusion/functions/index.html
+++ b/autoapi/datafusion/functions/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/index.html b/autoapi/datafusion/index.html
index f4c3d351..fd9864c3 100644
--- a/autoapi/datafusion/index.html
+++ b/autoapi/datafusion/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/input/base/index.html
b/autoapi/datafusion/input/base/index.html
index cf44b1ae..4d2835a8 100644
--- a/autoapi/datafusion/input/base/index.html
+++ b/autoapi/datafusion/input/base/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/input/index.html
b/autoapi/datafusion/input/index.html
index 896164da..fada80ae 100644
--- a/autoapi/datafusion/input/index.html
+++ b/autoapi/datafusion/input/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/input/location/index.html
b/autoapi/datafusion/input/location/index.html
index 9d466821..2e087e41 100644
--- a/autoapi/datafusion/input/location/index.html
+++ b/autoapi/datafusion/input/location/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/object_store/index.html
b/autoapi/datafusion/object_store/index.html
index 71ad4a83..e780cdf8 100644
--- a/autoapi/datafusion/object_store/index.html
+++ b/autoapi/datafusion/object_store/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/plan/index.html
b/autoapi/datafusion/plan/index.html
index ddf72b35..c5f958c6 100644
--- a/autoapi/datafusion/plan/index.html
+++ b/autoapi/datafusion/plan/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/record_batch/index.html
b/autoapi/datafusion/record_batch/index.html
index f322281d..7e5bdde2 100644
--- a/autoapi/datafusion/record_batch/index.html
+++ b/autoapi/datafusion/record_batch/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/substrait/index.html
b/autoapi/datafusion/substrait/index.html
index 9b71cde5..d91a000a 100644
--- a/autoapi/datafusion/substrait/index.html
+++ b/autoapi/datafusion/substrait/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/datafusion/udf/index.html
b/autoapi/datafusion/udf/index.html
index 5452e591..b129e3a1 100644
--- a/autoapi/datafusion/udf/index.html
+++ b/autoapi/datafusion/udf/index.html
@@ -232,6 +232,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/autoapi/index.html b/autoapi/index.html
index da449fcf..c9749203 100644
--- a/autoapi/index.html
+++ b/autoapi/index.html
@@ -35,7 +35,7 @@
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="datafusion" href="datafusion/index.html" />
- <link rel="prev" title="Introduction"
href="../contributor-guide/introduction.html" />
+ <link rel="prev" title="Python Extensions"
href="../contributor-guide/ffi.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -415,11 +420,11 @@
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link"
href="../contributor-guide/introduction.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="../contributor-guide/ffi.html"
title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Introduction</p>
+ <p class="prev-next-title">Python Extensions</p>
</div>
</a>
<a class='right-next' id="next-link" href="datafusion/index.html"
title="next page">
diff --git a/contributor-guide/ffi.html b/contributor-guide/ffi.html
new file mode 100644
index 00000000..2d3a3011
--- /dev/null
+++ b/contributor-guide/ffi.html
@@ -0,0 +1,639 @@
+<!DOCTYPE html>
+
+<html lang="en" data-content_root="../">
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
+
+ <title>Python Extensions — Apache Arrow DataFusion
documentation</title>
+
+ <link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+
+
+ <link rel="stylesheet"
+ href="../_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+
+
+
+
+
+ <link rel="stylesheet" type="text/css"
href="../_static/pygments.css?v=8f2a1f02" />
+ <link rel="stylesheet" type="text/css"
href="../_static/styles/pydata-sphinx-theme.css?v=1140d252" />
+ <link rel="stylesheet" type="text/css"
href="../_static/graphviz.css?v=4ae1632d" />
+ <link rel="stylesheet" type="text/css"
href="../_static/theme_overrides.css?v=dca7052a" />
+
+ <link rel="preload" as="script"
href="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+
+ <script src="../_static/documentation_options.js?v=8a448e45"></script>
+ <script src="../_static/doctools.js?v=9bcbadda"></script>
+ <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
+ <link rel="index" title="Index" href="../genindex.html" />
+ <link rel="search" title="Search" href="../search.html" />
+ <link rel="next" title="API Reference" href="../autoapi/index.html" />
+ <link rel="prev" title="Introduction" href="introduction.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <meta name="docsearch:language" content="en">
+
+
+ <!-- Google Analytics -->
+
+ </head>
+ <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
+
+ <div class="container-fluid" id="banner"></div>
+
+
+
+
+ <div class="container-xl">
+ <div class="row">
+
+
+ <!-- Only show if we have sidebars configured, else just a small
margin -->
+ <div class="col-12 col-md-3 bd-sidebar">
+ <div class="sidebar-start-items">
+<a class="navbar-brand" href="../index.html">
+ <img src="../_static/images/2x_bgwhite_original.png" class="logo" alt="logo">
+</a>
+
+<form class="bd-search d-flex align-items-center" action="../search.html"
method="get">
+ <i class="icon fas fa-search"></i>
+ <input type="search" class="form-control" name="q" id="search-input"
placeholder="Search the docs ..." aria-label="Search the docs ..."
autocomplete="off" >
+</form>
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-python";>
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://docs.rs/datafusion/latest/datafusion/";>
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion/blob/main/CODE_OF_CONDUCT.md";>
+ Code of conduct
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-python/tree/main/examples";>
+ Examples
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/data-sources.html">
+ Data Sources
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal"
href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1"
name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal"
href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal"
href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal"
href="../user-guide/common-operations/expressions.html">
+ Expressions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal"
href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal"
href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal"
href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal"
href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal"
href="../user-guide/common-operations/udf-and-udfa.html">
+ User-Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2"
name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/arrow.html">
+ Arrow
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/table_provider.html">
+ Custom Table Provider
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/configuration.html">
+ Configuration
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Python Extensions
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../autoapi/index.html">
+ API Reference
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-3"
name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../autoapi/datafusion/index.html">
+ datafusion
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-4"
name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal"
href="../autoapi/datafusion/catalog/index.html">
+ datafusion.catalog
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal"
href="../autoapi/datafusion/context/index.html">
+ datafusion.context
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal"
href="../autoapi/datafusion/dataframe/index.html">
+ datafusion.dataframe
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal"
href="../autoapi/datafusion/expr/index.html">
+ datafusion.expr
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal"
href="../autoapi/datafusion/functions/index.html">
+ datafusion.functions
+ </a>
+ </li>
+ <li class="toctree-l3 has-children">
+ <a class="reference internal"
href="../autoapi/datafusion/input/index.html">
+ datafusion.input
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-5"
name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l4">
+ <a class="reference internal"
href="../autoapi/datafusion/input/base/index.html">
+ datafusion.input.base
+ </a>
+ </li>
+ <li class="toctree-l4">
+ <a class="reference internal"
href="../autoapi/datafusion/input/location/index.html">
+ datafusion.input.location
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal"
href="../autoapi/datafusion/object_store/index.html">
+ datafusion.object_store
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal"
href="../autoapi/datafusion/plan/index.html">
+ datafusion.plan
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal"
href="../autoapi/datafusion/record_batch/index.html">
+ datafusion.record_batch
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal"
href="../autoapi/datafusion/substrait/index.html">
+ datafusion.substrait
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal"
href="../autoapi/datafusion/udf/index.html">
+ datafusion.udf
+ </a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+</ul>
+
+
+ </div>
+</nav>
+ </div>
+ <div class="sidebar-end-items">
+ </div>
+ </div>
+
+
+
+
+ <div class="d-none d-xl-block col-xl-2 bd-toc">
+
+
+ <div class="toc-item">
+
+<div class="tocsection onthispage pt-5 pb-3">
+ <i class="fas fa-list"></i> On this page
+</div>
+
+<nav id="bd-toc-nav">
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#the-primary-issue">
+ The Primary Issue
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#the-ffi-approach">
+ The FFI Approach
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#inspiration-from-arrow">
+ Inspiration from Arrow
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#implementation-details">
+ Implementation Details
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#alternative-approach">
+ Alternative Approach
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#status-of-work">
+ Status of Work
+ </a>
+ </li>
+</ul>
+
+</nav>
+ </div>
+
+ <div class="toc-item">
+
+ </div>
+
+
+ </div>
+
+
+
+
+
+
+ <main class="col-12 col-md-9 col-xl-7 py-md-5 pl-md-5 pr-md-4
bd-content" role="main">
+
+ <div>
+
+ <section id="python-extensions">
+<h1>Python Extensions<a class="headerlink" href="#python-extensions"
title="Link to this heading">¶</a></h1>
+<p>The DataFusion in Python project is designed to allow users to extend its
functionality in a few core
+areas. Ideally many users would like to package their extensions as a Python
package and easily
+integrate that package with this project. This page serves to describe some of
the challenges we face
+when doing these integrations and the approach our project uses.</p>
+<section id="the-primary-issue">
+<h2>The Primary Issue<a class="headerlink" href="#the-primary-issue"
title="Link to this heading">¶</a></h2>
+<p>Suppose you wish to use DataFusion and you have a custom data source that
can produce tables that
+can then be queried against, similar to how you can register a <a
class="reference internal" href="../user-guide/io/csv.html#io-csv"><span
class="std std-ref">CSV</span></a> or
+<a class="reference internal"
href="../user-guide/io/parquet.html#io-parquet"><span class="std
std-ref">Parquet</span></a> file. In DataFusion terminology, you likely want to
implement a
+<a class="reference internal"
href="../user-guide/io/table_provider.html#io-custom-table-provider"><span
class="std std-ref">Custom Table Provider</span></a>. In an effort to make your
data source
+as performant as possible and to utilize the features of DataFusion, you may
decide to write
+your source in Rust and then expose it through <a class="reference external"
href="https://pyo3.rs";>PyO3</a> as a Python library.</p>
+<p>At first glance, it may appear the best way to do this is to add the <code
class="docutils literal notranslate"><span
class="pre">datafusion-python</span></code>
+crate as a dependency, provide a <code class="docutils literal
notranslate"><span class="pre">PyTable</span></code>, and then to register it
with the
+<code class="docutils literal notranslate"><span
class="pre">SessionContext</span></code>. Unfortunately, this will not work.</p>
+<p>When you produce your code as a Python library and it needs to interact
with the DataFusion
+library, at the lowest level they communicate through an Application Binary
Interface (ABI).
+The acronym sounds similar to API (Application Programming Interface), but it
is distinctly
+different.</p>
+<p>The ABI sets the standard for how these libraries can share data and
functions between each
+other. One of the key differences between Rust and other programming languages
is that Rust
+does not have a stable ABI. What this means in practice is that if you compile
a Rust library
+with one version of the <code class="docutils literal notranslate"><span
class="pre">rustc</span></code> compiler and I compile another library to
interface with it
+but I use a different version of the compiler, there is no guarantee the
interface will be
+the same.</p>
+<p>In practice, this means that a Python library built with <code
class="docutils literal notranslate"><span
class="pre">datafusion-python</span></code> as a Rust
+dependency will generally <strong>not</strong> be compatible with the
DataFusion Python package, even
+if they reference the same version of <code class="docutils literal
notranslate"><span class="pre">datafusion-python</span></code>. If you attempt
to do this, it may
+work on your local computer if you have built both packages with the same
optimizations.
+This can sometimes lead to a false expectation that the code will work, but it
frequently
+breaks the moment you try to use your package against the released
packages.</p>
+<p>You can find more information about the Rust ABI in their
+<a class="reference external"
href="https://doc.rust-lang.org/reference/abi.html";>online
documentation</a>.</p>
+</section>
+<section id="the-ffi-approach">
+<h2>The FFI Approach<a class="headerlink" href="#the-ffi-approach" title="Link
to this heading">¶</a></h2>
+<p>Rust supports interacting with other programming languages through it’s
Foreign Function
+Interface (FFI). The advantage of using the FFI is that it enables you to
write data structures
+and functions that have a stable ABI. The allows you to use Rust code with C,
Python, and
+other languages. In fact, the <a class="reference external"
href="https://pyo3.rs";>PyO3</a> library uses the FFI to share data
+and functions between Python and Rust.</p>
+<p>The approach we are taking in the DataFusion in Python project is to
incrementally expose
+more portions of the DataFusion project via FFI interfaces. This allows users
to write Rust
+code that does <strong>not</strong> require the <code class="docutils literal
notranslate"><span class="pre">datafusion-python</span></code> crate as a
dependency, expose their
+code in Python via PyO3, and have it interact with the DataFusion Python
package.</p>
+<p>Early adopters of this approach include <a class="reference external"
href="https://delta-io.github.io/delta-rs/";>delta-rs</a>
+who has adapted their Table Provider for use in <code class="docutils literal
notranslate"><span class="pre">`datafusion-python`</span></code> with only a
few lines
+of code. Also, the DataFusion Python project uses the existing definitions from
+<a class="reference external"
href="https://arrow.apache.org/docs/format/CStreamInterface.html";>Apache Arrow
CStream Interface</a>
+to support importing <strong>and</strong> exporting tables. Any Python package
that supports reading
+the Arrow C Stream interface can work with DataFusion Python out of the box!
You can read
+more about working with Arrow sources in the <a class="reference internal"
href="../user-guide/data-sources.html#user-guide-data-sources"><span class="std
std-ref">Data Sources</span></a>
+page.</p>
+<p>To learn more about the Foreign Function Interface in Rust, the
+<a class="reference external"
href="https://doc.rust-lang.org/nomicon/ffi.html";>Rustonomicon</a> is a good
resource.</p>
+</section>
+<section id="inspiration-from-arrow">
+<h2>Inspiration from Arrow<a class="headerlink" href="#inspiration-from-arrow"
title="Link to this heading">¶</a></h2>
+<p>DataFusion is built upon <a class="reference external"
href="https://arrow.apache.org/";>Apache Arrow</a>. The canonical Python
+Arrow implementation, <a class="reference external"
href="https://arrow.apache.org/docs/python/index.html";>pyarrow</a> provides
+an excellent way to share Arrow data between Python projects without
performing any copy
+operations on the data. They do this by using a well defined set of
interfaces. You can
+find the details about their stream interface
+<a class="reference external"
href="https://arrow.apache.org/docs/format/CStreamInterface.html";>here</a>. The
+<a class="reference external" href="https://github.com/apache/arrow-rs";>Rust
Arrow Implementation</a> also supports these
+<code class="docutils literal notranslate"><span class="pre">C</span></code>
style definitions via the Foreign Function Interface.</p>
+<p>In addition to using these interfaces to transfer Arrow data between
libraries, <code class="docutils literal notranslate"><span
class="pre">pyarrow</span></code>
+goes one step further to make sharing the interfaces easier in Python. They do
this
+by exposing PyCapsules that contain the expected functionality.</p>
+<p>You can learn more about PyCapsules from the official
+<a class="reference external"
href="https://docs.python.org/3/c-api/capsule.html";>Python online
documentation</a>. PyCapsules
+have excellent support in PyO3 already. The
+<a class="reference external"
href="https://pyo3.rs/main/doc/pyo3/types/struct.pycapsule";>PyO3 online
documentation</a> is a good source
+for more details on using PyCapsules in Rust.</p>
+<p>Two lessons we leverage from the Arrow project in DataFusion Python are:</p>
+<ul class="simple">
+<li><p>We reuse the existing Arrow FFI functionality wherever
possible.</p></li>
+<li><p>We expose PyCapsules that contain a FFI stable struct.</p></li>
+</ul>
+</section>
+<section id="implementation-details">
+<h2>Implementation Details<a class="headerlink" href="#implementation-details"
title="Link to this heading">¶</a></h2>
+<p>The bulk of the code necessary to perform our FFI operations is in the
upstream
+<a class="reference external"
href="https://datafusion.apache.org/";>DataFusion</a> core repository. You can
review the code and
+documentation in the <a class="reference external"
href="https://crates.io/crates/datafusion-ffi";>datafusion-ffi</a> crate.</p>
+<p>Our FFI implementation is narrowly focused at sharing data and functions
with Rust backed
+libraries. This allows us to use the <a class="reference external"
href="https://crates.io/crates/abi_stable";>abi_stable crate</a>.
+This is an excellent crate that allows for easy conversion between Rust native
types
+and FFI-safe alternatives. For example, if you needed to pass a <code
class="docutils literal notranslate"><span
class="pre">Vec<String></span></code> via FFI,
+you can simply convert it to a <code class="docutils literal
notranslate"><span class="pre">RVec<RString></span></code> in an
intuitive manner. It also supports
+features like <code class="docutils literal notranslate"><span
class="pre">RResult</span></code> and <code class="docutils literal
notranslate"><span class="pre">ROption</span></code> that do not have an
obvious translation to a
+C equivalent.</p>
+<p>The <a class="reference external"
href="https://crates.io/crates/datafusion-ffi";>datafusion-ffi</a> crate has
been designed to make it easy to convert from DataFusion
+traits into their FFI counterparts. For example, if you have defined a custom
+<a class="reference external"
href="https://docs.rs/datafusion/45.0.0/datafusion/catalog/trait.TableProvider.html";>TableProvider</a>
+and you want to create a sharable FFI counterpart, you could write:</p>
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="kd">let</span><span class="w">
</span><span class="n">my_provider</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="n">MyTableProvider</span><span class="p">::</span><span
class="n">default</span><span class="p">();</span>
+<span class="kd">let</span><span class="w"> </span><span
class="n">ffi_provider</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="n">FFI_TableProvider</span><span class="p">::</span><span
class="n">new</span><span class="p">(</span><span class="n">Arc</span><span
class="p">::</span><span class="n">new</span><span class="p">(</span><span
class="n">my_provider</span><span class="p">),</span><span class="w">
</span><span class="kc">false</span><span [...]
+</pre></div>
+</div>
+<p>If you were interfacing with a library that provided the above <code
class="docutils literal notranslate"><span
class="pre">FFI_TableProvider</span></code> and
+you needed to turn it back into an <code class="docutils literal
notranslate"><span class="pre">TableProvider</span></code>, you can turn it
into a
+<code class="docutils literal notranslate"><span
class="pre">ForeignTableProvider</span></code> with implements the <code
class="docutils literal notranslate"><span
class="pre">TableProvider</span></code> trait.</p>
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="kd">let</span><span class="w">
</span><span class="n">foreign_provider</span><span class="p">:</span><span
class="w"> </span><span class="nc">ForeignTableProvider</span><span class="w">
</span><span class="o">=</span><span class="w"> </span><span
class="n">ffi_provider</span><span class="p">.</span><span
class="n">into</span><span class="p">();</span>
+</pre></div>
+</div>
+<p>If you review the code in <a class="reference external"
href="https://crates.io/crates/datafusion-ffi";>datafusion-ffi</a> you will find
that each of the traits we share
+across the boundary has two portions, one with a <code class="docutils literal
notranslate"><span class="pre">FFI_</span></code> prefix and one with a <code
class="docutils literal notranslate"><span class="pre">Foreign</span></code>
+prefix. This is used to distinguish which side of the FFI boundary that struct
is
+designed to be used on. The structures with the <code class="docutils literal
notranslate"><span class="pre">FFI_</span></code> prefix are to be used on the
+<strong>provider</strong> of the structure. In the example we’re showing, this
means the code that has
+written the underlying <code class="docutils literal notranslate"><span
class="pre">TableProvider</span></code> implementation to access your custom
data source.
+The structures with the <code class="docutils literal notranslate"><span
class="pre">Foreign</span></code> prefix are to be used by the receiver. In
this case,
+it is the <code class="docutils literal notranslate"><span
class="pre">datafusion-python</span></code> library.</p>
+<p>In order to share these FFI structures, we need to wrap them in some kind
of Python object
+that can be used to interface from one package to another. As described in the
above
+section on our inspiration from Arrow, we use <code class="docutils literal
notranslate"><span class="pre">PyCapsule</span></code>. We can create a <code
class="docutils literal notranslate"><span class="pre">PyCapsule</span></code>
+for our provider thusly:</p>
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="kd">let</span><span class="w">
</span><span class="n">name</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="n">CString</span><span
class="p">::</span><span class="n">new</span><span class="p">(</span><span
class="s">"datafusion_table_provider"</span><span
class="p">)</span><span class="o">?</span><span class="p">;</span>
+<span class="kd">let</span><span class="w"> </span><span
class="n">my_capsule</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="n">PyCapsule</span><span
class="p">::</span><span class="n">new_bound</span><span
class="p">(</span><span class="n">py</span><span class="p">,</span><span
class="w"> </span><span class="n">provider</span><span class="p">,</span><span
class="w"> </span><span class="nb">Some</span><span class="p">(</span><span
class="n">nam [...]
+</pre></div>
+</div>
+<p>On the receiving side, turn this pycapsule object into the <code
class="docutils literal notranslate"><span
class="pre">FFI_TableProvider</span></code>, which
+can then be turned into a <code class="docutils literal notranslate"><span
class="pre">ForeignTableProvider</span></code> the associated code is:</p>
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="kd">let</span><span class="w">
</span><span class="n">capsule</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="n">capsule</span><span
class="p">.</span><span class="n">downcast</span><span class="p">::</span><span
class="o"><</span><span class="n">PyCapsule</span><span
class="o">></span><span class="p">()</span><span class="o">?</span><span
class=" [...]
+<span class="kd">let</span><span class="w"> </span><span
class="n">provider</span><span class="w"> </span><span class="o">=</span><span
class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span
class="p">{</span><span class="w"> </span><span class="n">capsule</span><span
class="p">.</span><span class="n">reference</span><span
class="p">::</span><span class="o"><</span><span
class="n">FFI_TableProvider</span><span class="o">></span><span
class="p">()</span><span c [...]
+</pre></div>
+</div>
+<p>By convention the <code class="docutils literal notranslate"><span
class="pre">datafusion-python</span></code> library expects a Python object
that has a
+<code class="docutils literal notranslate"><span
class="pre">TableProvider</span></code> PyCapsule to have this capsule
accessible by calling a function named
+<code class="docutils literal notranslate"><span
class="pre">__datafusion_table_provider__</span></code>. You can see a complete
working example of how to
+share a <code class="docutils literal notranslate"><span
class="pre">TableProvider</span></code> from one python library to DataFusion
Python in the
+<a class="reference external"
href="https://github.com/apache/datafusion-python/tree/main/examples/ffi-table-provider";>repository
examples folder</a>.</p>
+<p>This section has been written using <code class="docutils literal
notranslate"><span class="pre">TableProvider</span></code> as an example. It is
the first
+extension that has been written using this approach and the most thoroughly
implemented.
+As we continue to expose more of the DataFusion features, we intend to follow
this same
+design pattern.</p>
+</section>
+<section id="alternative-approach">
+<h2>Alternative Approach<a class="headerlink" href="#alternative-approach"
title="Link to this heading">¶</a></h2>
+<p>Suppose you needed to expose some other features of DataFusion and you
could not wait
+for the upstream repository to implement the FFI approach we describe. In this
case
+you decide to create your dependency on the <code class="docutils literal
notranslate"><span class="pre">datafusion-python</span></code> crate
instead.</p>
+<p>As we discussed, this is not guaranteed to work across different compiler
versions and
+optimization levels. If you wish to go down this route, there are two
approaches we
+have identified you can use.</p>
+<ol class="arabic simple">
+<li><p>Re-export all of <code class="docutils literal notranslate"><span
class="pre">datafusion-python</span></code> yourself with your extensions built
in.</p></li>
+<li><p>Carefully synchonize your software releases with the <code
class="docutils literal notranslate"><span
class="pre">datafusion-python</span></code> CI build
+system so that your libraries use the exact same compiler, features, and
+optimization level.</p></li>
+</ol>
+<p>We currently do not recommend either of these approaches as they are
difficult to
+maintain over a long period. Additionally, they require a tight version
coupling
+between libraries.</p>
+</section>
+<section id="status-of-work">
+<h2>Status of Work<a class="headerlink" href="#status-of-work" title="Link to
this heading">¶</a></h2>
+<p>At the time of this writing, the FFI features are under active development.
To see
+the latest status, we recommend reviewing the code in the <a class="reference
external" href="https://crates.io/crates/datafusion-ffi";>datafusion-ffi</a>
crate.</p>
+</section>
+</section>
+
+
+ </div>
+
+
+ <!-- Previous / next buttons -->
+<div class='prev-next-area'>
+ <a class='left-prev' id="prev-link" href="introduction.html"
title="previous page">
+ <i class="fas fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">Introduction</p>
+ </div>
+ </a>
+ <a class='right-next' id="next-link" href="../autoapi/index.html"
title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">API Reference</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
+</div>
+
+ </main>
+
+
+ </div>
+ </div>
+
+ <script
src="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+
+<!-- Based on pydata_sphinx_theme/footer.html -->
+<footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p class="copyright">
+ © Copyright 2019-2024, Apache Software Foundation.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p class="sphinx-version">
+Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 8.1.3.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p>Apache Arrow DataFusion, Arrow DataFusion, Apache, the Apache feather
logo, and the Apache Arrow DataFusion project logo</p>
+ <p>are either registered trademarks or trademarks of The Apache Software
Foundation in the United States and other countries.</p>
+ </div>
+ </div>
+</footer>
+
+
+ </body>
+</html>
\ No newline at end of file
diff --git a/contributor-guide/introduction.html
b/contributor-guide/introduction.html
index 1ff78cef..883463d9 100644
--- a/contributor-guide/introduction.html
+++ b/contributor-guide/introduction.html
@@ -34,7 +34,7 @@
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="API Reference" href="../autoapi/index.html" />
+ <link rel="next" title="Python Extensions" href="ffi.html" />
<link rel="prev" title="SQL" href="../user-guide/sql.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -528,10 +533,10 @@ to your settings. You can find the appropriate rust flags
by looking in the
<p class="prev-next-title">SQL</p>
</div>
</a>
- <a class='right-next' id="next-link" href="../autoapi/index.html"
title="next page">
+ <a class='right-next' id="next-link" href="ffi.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">API Reference</p>
+ <p class="prev-next-title">Python Extensions</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/genindex.html b/genindex.html
index c65d715c..3f626d6e 100644
--- a/genindex.html
+++ b/genindex.html
@@ -230,6 +230,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/index.html b/index.html
index 57890523..9a935a6c 100644
--- a/index.html
+++ b/index.html
@@ -232,6 +232,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/objects.inv b/objects.inv
index a9e2b001..7d58068e 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/py-modindex.html b/py-modindex.html
index 23fde1c9..48b97dc1 100644
--- a/py-modindex.html
+++ b/py-modindex.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/search.html b/search.html
index acc066c4..b7bc90c9 100644
--- a/search.html
+++ b/search.html
@@ -237,6 +237,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/searchindex.js b/searchindex.js
index 55bc5567..a04d6be5 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"API Reference": [[14, null]], "Aggregate
Functions": [[18, "aggregate-functions"], [25, "aggregate-functions"], [26,
"aggregate-functions"]], "Aggregation": [[18, null]], "Arrays": [[20,
"arrays"]], "Arrow": [[30, null]], "Attributes": [[3, "attributes"], [5,
"attributes"], [9, "attributes"], [13, "attributes"]], "Available Functions":
[[26, "available-functions"]], "Avro": [[31, null]], "Basic Operations": [[19,
null]], "Boolean": [[20, "boolean"]], "CSV" [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"API Reference": [[14, null]], "Aggregate
Functions": [[19, "aggregate-functions"], [26, "aggregate-functions"], [27,
"aggregate-functions"]], "Aggregation": [[19, null]], "Alternative Approach":
[[15, "alternative-approach"]], "Arrays": [[21, "arrays"]], "Arrow": [[31,
null]], "Attributes": [[3, "attributes"], [5, "attributes"], [9, "attributes"],
[13, "attributes"]], "Available Functions": [[27, "available-functions"]],
"Avro": [[32, null]], "Basic Operat [...]
\ No newline at end of file
diff --git a/user-guide/basics.html b/user-guide/basics.html
index b7d61220..7e4b40e8 100644
--- a/user-guide/basics.html
+++ b/user-guide/basics.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/common-operations/aggregations.html
b/user-guide/common-operations/aggregations.html
index 7f0221d5..5cde9abc 100644
--- a/user-guide/common-operations/aggregations.html
+++ b/user-guide/common-operations/aggregations.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -488,20 +493,20 @@ For grouping the <code class="code docutils literal
notranslate"><span class="pr
<span class="o">....</span><span class="p">:</span>
<span class="n">Out</span><span class="p">[</span><span
class="mi">10</span><span class="p">]:</span>
<span class="n">DataFrame</span><span class="p">()</span>
-<span
class="o">+----------+--------+-----------+--------------------+-----------+</span>
-<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span>
<span class="o">|</span> <span class="n">Type</span> <span
class="mi">2</span> <span class="o">|</span> <span class="n">Max</span> <span
class="n">Speed</span> <span class="o">|</span> <span class="n">Avg</span>
<span class="n">Speed</span> <span class="o">|</span> <span
class="n">Min</span> <span class="n">Speed</span> <span class="o">|</span>
-<span
class="o">+----------+--------+-----------+--------------------+-----------+</span>
-<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="o">|</span> <span class="mi">105</span>
<span class="o">|</span> <span class="mf">81.8</span> <span
class="o">|</span> <span class="mi">60</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="o">|</span> <span class="mi">90</span>
<span class="o">|</span> <span class="mf">68.05263157894737</span> <span
class="o">|</span> <span class="mi">40</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Ground</span> <span
class="o">|</span> <span class="o">|</span> <span class="mi">120</span>
<span class="o">|</span> <span class="mf">66.66666666666667</span> <span
class="o">|</span> <span class="mi">35</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="n">Fairy</span> <span class="o">|</span> <span
class="mi">45</span> <span class="o">|</span> <span
class="mf">32.5</span> <span class="o">|</span> <span
class="mi">20</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Fighting</span> <span
class="o">|</span> <span class="o">|</span> <span class="mi">95</span>
<span class="o">|</span> <span class="mf">66.14285714285714</span> <span
class="o">|</span> <span class="mi">35</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Psychic</span> <span
class="o">|</span> <span class="o">|</span> <span class="mi">150</span>
<span class="o">|</span> <span class="mf">100.57142857142857</span> <span
class="o">|</span> <span class="mi">42</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="o">|</span> <span class="mi">60</span>
<span class="o">|</span> <span class="mf">60.0</span> <span
class="o">|</span> <span class="mi">60</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Dragon</span> <span
class="o">|</span> <span class="o">|</span> <span class="mi">70</span>
<span class="o">|</span> <span class="mf">60.0</span> <span
class="o">|</span> <span class="mi">50</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span
class="mi">80</span> <span class="o">|</span> <span
class="mf">55.0</span> <span class="o">|</span> <span
class="mi">30</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="n">Flying</span> <span class="o">|</span> <span
class="mi">105</span> <span class="o">|</span> <span
class="mf">93.33333333333333</span> <span class="o">|</span> <span
class="mi">70</span> <span class="o">|</span>
-<span
class="o">+----------+--------+-----------+--------------------+-----------+</span>
+<span
class="o">+----------+--------+-----------+-------------------+-----------+</span>
+<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span>
<span class="o">|</span> <span class="n">Type</span> <span
class="mi">2</span> <span class="o">|</span> <span class="n">Max</span> <span
class="n">Speed</span> <span class="o">|</span> <span class="n">Avg</span>
<span class="n">Speed</span> <span class="o">|</span> <span
class="n">Min</span> <span class="n">Speed</span> <span class="o">|</span>
+<span
class="o">+----------+--------+-----------+-------------------+-----------+</span>
+<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span
class="mi">80</span> <span class="o">|</span> <span
class="mf">55.0</span> <span class="o">|</span> <span
class="mi">30</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="n">Flying</span> <span class="o">|</span> <span
class="mi">105</span> <span class="o">|</span> <span
class="mf">93.33333333333333</span> <span class="o">|</span> <span
class="mi">70</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="n">Ground</span> <span class="o">|</span> <span
class="mi">85</span> <span class="o">|</span> <span
class="mf">80.5</span> <span class="o">|</span> <span
class="mi">76</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="n">Flying</span> <span class="o">|</span> <span
class="mi">90</span> <span class="o">|</span> <span
class="mf">72.5</span> <span class="o">|</span> <span
class="mi">55</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span
class="mi">30</span> <span class="o">|</span> <span
class="mf">27.5</span> <span class="o">|</span> <span
class="mi">25</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span
class="mi">100</span> <span class="o">|</span> <span
class="mf">85.0</span> <span class="o">|</span> <span
class="mi">70</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="n">Steel</span> <span class="o">|</span> <span
class="mi">70</span> <span class="o">|</span> <span
class="mf">57.5</span> <span class="o">|</span> <span
class="mi">45</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Ghost</span> <span
class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span
class="mi">130</span> <span class="o">|</span> <span
class="mf">103.75</span> <span class="o">|</span> <span
class="mi">80</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Psychic</span> <span
class="o">|</span> <span class="n">Fairy</span> <span class="o">|</span> <span
class="mi">90</span> <span class="o">|</span> <span
class="mf">90.0</span> <span class="o">|</span> <span
class="mi">90</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="n">Flying</span> <span class="o">|</span> <span
class="mi">100</span> <span class="o">|</span> <span
class="mf">100.0</span> <span class="o">|</span> <span
class="mi">100</span> <span class="o">|</span>
+<span
class="o">+----------+--------+-----------+-------------------+-----------+</span>
</pre></div>
</div>
<section id="setting-parameters">
@@ -559,15 +564,15 @@ Pokemon set. Since there will be many entries of <code
class="docutils literal n
<span
class="o">+----------+--------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span>
<span class="o">|</span> <span class="n">Type</span> <span
class="mi">2</span> <span class="n">List</span>
<span class="o">|</span>
<span
class="o">+----------+--------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Fairy</span><span class="p">,</span> <span
class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Fairy</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Grass</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span
class="p">,</span> <span class="n">Poison</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Flying</span><span
class="p">,</span> <span class="n">Ground</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Poison</span><span
class="p">,</span> <span class="n">Grass</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">]</span> <span
class="o">|</span>
+<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Ground</span><span
class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Steel</span><span
class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Fairy</span> <span
class="o">|</span> <span class="p">[]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span
class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Flying</span><span
class="p">,</span> <span class="n">Dragon</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Dark</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Ice</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span
class="n">Fighting</span><span class="p">,</span> <span
class="n">Psychic</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Poison</span><span
class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Dragon</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span
class="n">Poison</span><span class="p">,</span> <span class="p">,</span> <span
class="n">Ice</span><span class="p">,</span> <span class="n">Dark</span><span
class="p">,</span> <span class="n">Fighting</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Ground</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Rock</span><span
class="p">]</span> <span
class="o">|</span>
<span
class="o">+----------+--------------------------------------------------+</span>
</pre></div>
@@ -582,15 +587,15 @@ entries entirely removed. The second is we can use the
<code class="docutils lit
<span
class="o">+----------+------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span>
<span class="o">|</span> <span class="n">Type</span> <span
class="mi">2</span> <span class="n">List</span>
<span class="o">|</span>
<span
class="o">+----------+------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Grass</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">]</span> <span
class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fairy</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span
class="n">Grass</span><span class="p">]</span> <span
class="o">|</span>
<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Ground</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Steel</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fairy</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Dragon</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span
class="n">Psychic</span><span class="p">,</span> <span
class="n">Fighting</span><span class="p">,</span> <span
class="n">Dark</span><span class="p">,</span> <span
class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Rock</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Ground</span><span class="p">,</span> <span
class="n">Water</span><span class="p">]</span> <span
class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Dragon</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fighting</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span
class="n">Ice</span><span class="p">,</span> <span class="n">Dark</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span
class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Rock</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Water</span><span
class="p">,</span> <span class="n">Ground</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">]</span> <span
class="o">|</span>
<span class="o">|</span> <span class="n">Ghost</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">]</span> <span
class="o">|</span>
<span
class="o">+----------+------------------------------------------------+</span>
@@ -600,15 +605,15 @@ entries entirely removed. The second is we can use the
<code class="docutils lit
<span
class="o">+----------+------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span>
<span class="o">|</span> <span class="n">Type</span> <span
class="mi">2</span> <span class="n">List</span>
<span class="o">|</span>
<span
class="o">+----------+------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Fairy</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Grass</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span
class="n">Poison</span><span class="p">]</span> <span
class="o">|</span>
-<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Ground</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fairy</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span
class="n">Grass</span><span class="p">]</span> <span
class="o">|</span>
+<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Steel</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Fairy</span> <span
class="o">|</span> <span
class="o">|</span>
<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Dragon</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Ice</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span
class="n">Fighting</span><span class="p">,</span> <span
class="n">Dark</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">,</span> <span
class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fighting</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span
class="n">Psychic</span><span class="p">,</span> <span
class="n">Ice</span><span class="p">,</span> <span class="n">Dark</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span
class="o">|</span>
<span class="o">|</span> <span class="n">Ground</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Rock</span><span
class="p">]</span> <span
class="o">|</span>
<span
class="o">+----------+------------------------------------------------+</span>
</pre></div>
diff --git a/user-guide/common-operations/basic-info.html
b/user-guide/common-operations/basic-info.html
index cba290a0..0d17396f 100644
--- a/user-guide/common-operations/basic-info.html
+++ b/user-guide/common-operations/basic-info.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -399,11 +404,11 @@
<span class="o">+-----+---------+--------+--------+</span>
<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span>
<span class="n">names</span> <span class="o">|</span> <span
class="n">random</span> <span class="o">|</span> <span class="n">groups</span>
<span class="o">|</span>
<span class="o">+-----+---------+--------+--------+</span>
-<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
<span class="n">python</span> <span class="o">|</span> <span
class="mi">617</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
<span class="n">ruby</span> <span class="o">|</span> <span
class="mi">82</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span>
<span class="n">java</span> <span class="o">|</span> <span
class="mi">137</span> <span class="o">|</span> <span class="n">B</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span>
<span class="n">haskell</span> <span class="o">|</span> <span
class="mi">183</span> <span class="o">|</span> <span class="n">C</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span>
<span class="n">go</span> <span class="o">|</span> <span
class="mi">244</span> <span class="o">|</span> <span class="n">B</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
<span class="n">python</span> <span class="o">|</span> <span
class="mi">747</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
<span class="n">ruby</span> <span class="o">|</span> <span
class="mi">57</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span>
<span class="n">java</span> <span class="o">|</span> <span
class="mi">376</span> <span class="o">|</span> <span class="n">B</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span>
<span class="n">haskell</span> <span class="o">|</span> <span
class="mi">591</span> <span class="o">|</span> <span class="n">C</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span>
<span class="n">go</span> <span class="o">|</span> <span
class="mi">123</span> <span class="o">|</span> <span class="n">B</span>
<span class="o">|</span>
<span class="o">+-----+---------+--------+--------+</span>
</pre></div>
</div>
@@ -414,8 +419,8 @@
<span class="o">+-----+--------+--------+--------+</span>
<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span>
<span class="n">names</span> <span class="o">|</span> <span
class="n">random</span> <span class="o">|</span> <span class="n">groups</span>
<span class="o">|</span>
<span class="o">+-----+--------+--------+--------+</span>
-<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
<span class="n">python</span> <span class="o">|</span> <span
class="mi">617</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
<span class="n">ruby</span> <span class="o">|</span> <span
class="mi">82</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
<span class="n">python</span> <span class="o">|</span> <span
class="mi">747</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
<span class="n">ruby</span> <span class="o">|</span> <span
class="mi">57</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
<span class="o">+-----+--------+--------+--------+</span>
</pre></div>
</div>
@@ -433,28 +438,28 @@ passing them to an Arrow table, and then converting them
to a pandas DataFrame.<
<div class="highlight-ipython notranslate"><div
class="highlight"><pre><span></span><span class="n">In</span> <span
class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span
class="n">df</span><span class="o">.</span><span
class="n">to_pandas</span><span class="p">()</span>
<span class="n">Out</span><span class="p">[</span><span
class="mi">8</span><span class="p">]:</span>
<span class="n">nrs</span> <span class="n">names</span> <span
class="n">random</span> <span class="n">groups</span>
-<span class="mi">0</span> <span class="mi">1</span> <span
class="n">python</span> <span class="mi">617</span> <span
class="n">A</span>
-<span class="mi">1</span> <span class="mi">2</span> <span
class="n">ruby</span> <span class="mi">82</span> <span
class="n">A</span>
-<span class="mi">2</span> <span class="mi">3</span> <span
class="n">java</span> <span class="mi">137</span> <span
class="n">B</span>
-<span class="mi">3</span> <span class="mi">4</span> <span
class="n">haskell</span> <span class="mi">183</span> <span
class="n">C</span>
-<span class="mi">4</span> <span class="mi">5</span> <span
class="n">go</span> <span class="mi">244</span> <span
class="n">B</span>
+<span class="mi">0</span> <span class="mi">1</span> <span
class="n">python</span> <span class="mi">747</span> <span
class="n">A</span>
+<span class="mi">1</span> <span class="mi">2</span> <span
class="n">ruby</span> <span class="mi">57</span> <span
class="n">A</span>
+<span class="mi">2</span> <span class="mi">3</span> <span
class="n">java</span> <span class="mi">376</span> <span
class="n">B</span>
+<span class="mi">3</span> <span class="mi">4</span> <span
class="n">haskell</span> <span class="mi">591</span> <span
class="n">C</span>
+<span class="mi">4</span> <span class="mi">5</span> <span
class="n">go</span> <span class="mi">123</span> <span
class="n">B</span>
</pre></div>
</div>
<p><a class="reference internal"
href="../../autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.describe"
title="datafusion.dataframe.DataFrame.describe"><code class="xref py py-func
docutils literal notranslate"><span class="pre">describe()</span></code></a>
shows a quick statistic summary of your data:</p>
<div class="highlight-ipython notranslate"><div
class="highlight"><pre><span></span><span class="n">In</span> <span
class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span
class="n">df</span><span class="o">.</span><span class="n">describe</span><span
class="p">()</span>
<span class="n">Out</span><span class="p">[</span><span
class="mi">9</span><span class="p">]:</span>
<span class="n">DataFrame</span><span class="p">()</span>
-<span
class="o">+------------+--------------------+-------+--------------------+--------+</span>
-<span class="o">|</span> <span class="n">describe</span> <span
class="o">|</span> <span class="n">nrs</span> <span
class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span
class="n">random</span> <span class="o">|</span> <span
class="n">groups</span> <span class="o">|</span>
-<span
class="o">+------------+--------------------+-------+--------------------+--------+</span>
-<span class="o">|</span> <span class="n">count</span> <span
class="o">|</span> <span class="mf">5.0</span> <span
class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span
class="mf">5.0</span> <span class="o">|</span> <span
class="mi">5</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">null_count</span> <span
class="o">|</span> <span class="mf">0.0</span> <span
class="o">|</span> <span class="mi">0</span> <span class="o">|</span> <span
class="mf">0.0</span> <span class="o">|</span> <span
class="mi">0</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">mean</span> <span
class="o">|</span> <span class="mf">3.0</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">252.6</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">std</span> <span
class="o">|</span> <span class="mf">1.5811388300841898</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">212.23406889564174</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
-<span class="o">|</span> <span class="nb">min</span> <span
class="o">|</span> <span class="mf">1.0</span> <span
class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span
class="mf">82.0</span> <span class="o">|</span> <span
class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="nb">max</span> <span
class="o">|</span> <span class="mf">5.0</span> <span
class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span
class="mf">617.0</span> <span class="o">|</span> <span
class="n">C</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">median</span> <span
class="o">|</span> <span class="mf">3.0</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">183.0</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
-<span
class="o">+------------+--------------------+-------+--------------------+--------+</span>
+<span
class="o">+------------+--------------------+-------+-------------------+--------+</span>
+<span class="o">|</span> <span class="n">describe</span> <span
class="o">|</span> <span class="n">nrs</span> <span
class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span
class="n">random</span> <span class="o">|</span> <span
class="n">groups</span> <span class="o">|</span>
+<span
class="o">+------------+--------------------+-------+-------------------+--------+</span>
+<span class="o">|</span> <span class="n">count</span> <span
class="o">|</span> <span class="mf">5.0</span> <span
class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span
class="mf">5.0</span> <span class="o">|</span> <span
class="mi">5</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">null_count</span> <span
class="o">|</span> <span class="mf">0.0</span> <span
class="o">|</span> <span class="mi">0</span> <span class="o">|</span> <span
class="mf">0.0</span> <span class="o">|</span> <span
class="mi">0</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">mean</span> <span
class="o">|</span> <span class="mf">3.0</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">378.8</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">std</span> <span
class="o">|</span> <span class="mf">1.5811388300841898</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">295.6335569586105</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">min</span> <span
class="o">|</span> <span class="mf">1.0</span> <span
class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span
class="mf">57.0</span> <span class="o">|</span> <span
class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">max</span> <span
class="o">|</span> <span class="mf">5.0</span> <span
class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span
class="mf">747.0</span> <span class="o">|</span> <span
class="n">C</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">median</span> <span
class="o">|</span> <span class="mf">3.0</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">376.0</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
+<span
class="o">+------------+--------------------+-------+-------------------+--------+</span>
</pre></div>
</div>
</section>
diff --git a/user-guide/common-operations/expressions.html
b/user-guide/common-operations/expressions.html
index 6542c9e6..3c6b44f2 100644
--- a/user-guide/common-operations/expressions.html
+++ b/user-guide/common-operations/expressions.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/common-operations/functions.html
b/user-guide/common-operations/functions.html
index d20ed611..fa774f0c 100644
--- a/user-guide/common-operations/functions.html
+++ b/user-guide/common-operations/functions.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -490,16 +495,16 @@ In here we will cover some of the more popular use cases.
If you want to view al
<span class="o">+--------------------------------+</span>
<span class="o">|</span> <span class="n">now</span><span class="p">()</span>
<span class="o">|</span>
<span class="o">+--------------------------------+</span>
-<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">20</span><span class="n">T01</span><span class="p">:</span><span
class="mi">55</span><span class="p">:</span><span
class="mf">24.583282540</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">20</span><span class="n">T01</span><span class="p">:</span><span
class="mi">55</span><span class="p">:</span><span
class="mf">24.583282540</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">20</span><span class="n">T01</span><span class="p">:</span><span
class="mi">55</span><span class="p">:</span><span
class="mf">24.583282540</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">20</span><span class="n">T01</span><span class="p">:</span><span
class="mi">55</span><span class="p">:</span><span
class="mf">24.583282540</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">20</span><span class="n">T01</span><span class="p">:</span><span
class="mi">55</span><span class="p">:</span><span
class="mf">24.583282540</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">20</span><span class="n">T01</span><span class="p">:</span><span
class="mi">55</span><span class="p">:</span><span
class="mf">24.583282540</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">20</span><span class="n">T01</span><span class="p">:</span><span
class="mi">55</span><span class="p">:</span><span
class="mf">24.583282540</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">20</span><span class="n">T01</span><span class="p">:</span><span
class="mi">55</span><span class="p">:</span><span
class="mf">24.583282540</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">20</span><span class="n">T01</span><span class="p">:</span><span
class="mi">55</span><span class="p">:</span><span
class="mf">24.583282540</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">20</span><span class="n">T01</span><span class="p">:</span><span
class="mi">55</span><span class="p">:</span><span
class="mf">24.583282540</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">21</span><span class="n">T21</span><span class="p">:</span><span
class="mi">08</span><span class="p">:</span><span
class="mf">42.620547788</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">21</span><span class="n">T21</span><span class="p">:</span><span
class="mi">08</span><span class="p">:</span><span
class="mf">42.620547788</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">21</span><span class="n">T21</span><span class="p">:</span><span
class="mi">08</span><span class="p">:</span><span
class="mf">42.620547788</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">21</span><span class="n">T21</span><span class="p">:</span><span
class="mi">08</span><span class="p">:</span><span
class="mf">42.620547788</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">21</span><span class="n">T21</span><span class="p">:</span><span
class="mi">08</span><span class="p">:</span><span
class="mf">42.620547788</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">21</span><span class="n">T21</span><span class="p">:</span><span
class="mi">08</span><span class="p">:</span><span
class="mf">42.620547788</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">21</span><span class="n">T21</span><span class="p">:</span><span
class="mi">08</span><span class="p">:</span><span
class="mf">42.620547788</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">21</span><span class="n">T21</span><span class="p">:</span><span
class="mi">08</span><span class="p">:</span><span
class="mf">42.620547788</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">21</span><span class="n">T21</span><span class="p">:</span><span
class="mi">08</span><span class="p">:</span><span
class="mf">42.620547788</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span
class="o">-</span><span class="mi">02</span><span class="o">-</span><span
class="mi">21</span><span class="n">T21</span><span class="p">:</span><span
class="mi">08</span><span class="p">:</span><span
class="mf">42.620547788</span><span class="n">Z</span> <span class="o">|</span>
<span class="o">+--------------------------------+</span>
</pre></div>
</div>
diff --git a/user-guide/common-operations/index.html
b/user-guide/common-operations/index.html
index e66669a2..32caf1f0 100644
--- a/user-guide/common-operations/index.html
+++ b/user-guide/common-operations/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/common-operations/joins.html
b/user-guide/common-operations/joins.html
index 026f289c..ba14264e 100644
--- a/user-guide/common-operations/joins.html
+++ b/user-guide/common-operations/joins.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/common-operations/select-and-filter.html
b/user-guide/common-operations/select-and-filter.html
index 63ebfb15..86ee7d18 100644
--- a/user-guide/common-operations/select-and-filter.html
+++ b/user-guide/common-operations/select-and-filter.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/common-operations/udf-and-udfa.html
b/user-guide/common-operations/udf-and-udfa.html
index 38b15b12..d4e37cc3 100644
--- a/user-guide/common-operations/udf-and-udfa.html
+++ b/user-guide/common-operations/udf-and-udfa.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/common-operations/windows.html
b/user-guide/common-operations/windows.html
index a2b08de4..cf976287 100644
--- a/user-guide/common-operations/windows.html
+++ b/user-guide/common-operations/windows.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -658,20 +663,20 @@ power in its <code class="docutils literal
notranslate"><span class="pre">"
<span class="o">....</span><span class="p">:</span>
<span class="n">Out</span><span class="p">[</span><span
class="mi">13</span><span class="p">]:</span>
<span class="n">DataFrame</span><span class="p">()</span>
-<span
class="o">+-----------------------+--------+--------+-------------------+</span>
-<span class="o">|</span> <span class="n">Name</span> <span
class="o">|</span> <span class="n">Attack</span> <span class="o">|</span> <span
class="n">Type</span> <span class="mi">1</span> <span class="o">|</span> <span
class="n">Average</span> <span class="n">Attack</span> <span
class="o">|</span>
-<span
class="o">+-----------------------+--------+--------+-------------------+</span>
-<span class="o">|</span> <span class="n">Metapod</span> <span
class="o">|</span> <span class="mi">20</span> <span class="o">|</span>
<span class="n">Bug</span> <span class="o">|</span> <span
class="mf">76.42857142857143</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Butterfree</span> <span
class="o">|</span> <span class="mi">45</span> <span class="o">|</span>
<span class="n">Bug</span> <span class="o">|</span> <span
class="mf">76.42857142857143</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Weedle</span> <span
class="o">|</span> <span class="mi">35</span> <span class="o">|</span>
<span class="n">Bug</span> <span class="o">|</span> <span
class="mf">76.42857142857143</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Kakuna</span> <span
class="o">|</span> <span class="mi">25</span> <span class="o">|</span>
<span class="n">Bug</span> <span class="o">|</span> <span
class="mf">76.42857142857143</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Beedrill</span> <span
class="o">|</span> <span class="mi">90</span> <span class="o">|</span>
<span class="n">Bug</span> <span class="o">|</span> <span
class="mf">76.42857142857143</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">BeedrillMega</span> <span
class="n">Beedrill</span> <span class="o">|</span> <span class="mi">150</span>
<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="mf">76.42857142857143</span> <span
class="o">|</span>
-<span class="o">|</span> <span class="n">Paras</span> <span
class="o">|</span> <span class="mi">70</span> <span class="o">|</span>
<span class="n">Bug</span> <span class="o">|</span> <span
class="mf">76.42857142857143</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Parasect</span> <span
class="o">|</span> <span class="mi">95</span> <span class="o">|</span>
<span class="n">Bug</span> <span class="o">|</span> <span
class="mf">76.42857142857143</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Venonat</span> <span
class="o">|</span> <span class="mi">55</span> <span class="o">|</span>
<span class="n">Bug</span> <span class="o">|</span> <span
class="mf">76.42857142857143</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Venomoth</span> <span
class="o">|</span> <span class="mi">65</span> <span class="o">|</span>
<span class="n">Bug</span> <span class="o">|</span> <span
class="mf">76.42857142857143</span> <span class="o">|</span>
-<span
class="o">+-----------------------+--------+--------+-------------------+</span>
+<span
class="o">+---------------------+--------+--------+----------------+</span>
+<span class="o">|</span> <span class="n">Name</span> <span
class="o">|</span> <span class="n">Attack</span> <span class="o">|</span> <span
class="n">Type</span> <span class="mi">1</span> <span class="o">|</span> <span
class="n">Average</span> <span class="n">Attack</span> <span class="o">|</span>
+<span
class="o">+---------------------+--------+--------+----------------+</span>
+<span class="o">|</span> <span class="n">Jynx</span> <span
class="o">|</span> <span class="mi">50</span> <span class="o">|</span>
<span class="n">Ice</span> <span class="o">|</span> <span
class="mf">67.5</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Articuno</span> <span
class="o">|</span> <span class="mi">85</span> <span class="o">|</span>
<span class="n">Ice</span> <span class="o">|</span> <span
class="mf">67.5</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Pidgey</span> <span
class="o">|</span> <span class="mi">45</span> <span class="o">|</span>
<span class="n">Normal</span> <span class="o">|</span> <span
class="mf">70.625</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Pidgeotto</span> <span
class="o">|</span> <span class="mi">60</span> <span class="o">|</span>
<span class="n">Normal</span> <span class="o">|</span> <span
class="mf">70.625</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Pidgeot</span> <span
class="o">|</span> <span class="mi">80</span> <span class="o">|</span>
<span class="n">Normal</span> <span class="o">|</span> <span
class="mf">70.625</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">PidgeotMega</span> <span
class="n">Pidgeot</span> <span class="o">|</span> <span class="mi">80</span>
<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="mf">70.625</span> <span
class="o">|</span>
+<span class="o">|</span> <span class="n">Rattata</span> <span
class="o">|</span> <span class="mi">56</span> <span class="o">|</span>
<span class="n">Normal</span> <span class="o">|</span> <span
class="mf">70.625</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Raticate</span> <span
class="o">|</span> <span class="mi">81</span> <span class="o">|</span>
<span class="n">Normal</span> <span class="o">|</span> <span
class="mf">70.625</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Spearow</span> <span
class="o">|</span> <span class="mi">60</span> <span class="o">|</span>
<span class="n">Normal</span> <span class="o">|</span> <span
class="mf">70.625</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fearow</span> <span
class="o">|</span> <span class="mi">90</span> <span class="o">|</span>
<span class="n">Normal</span> <span class="o">|</span> <span
class="mf">70.625</span> <span class="o">|</span>
+<span
class="o">+---------------------+--------+--------+----------------+</span>
</pre></div>
</div>
</section>
diff --git a/user-guide/configuration.html b/user-guide/configuration.html
index 257958d4..8ea8c390 100644
--- a/user-guide/configuration.html
+++ b/user-guide/configuration.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/data-sources.html b/user-guide/data-sources.html
index 6fb95668..0c7da12f 100644
--- a/user-guide/data-sources.html
+++ b/user-guide/data-sources.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/introduction.html b/user-guide/introduction.html
index 17f1c218..2da3a880 100644
--- a/user-guide/introduction.html
+++ b/user-guide/introduction.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/io/arrow.html b/user-guide/io/arrow.html
index 4fd6b96a..3016c379 100644
--- a/user-guide/io/arrow.html
+++ b/user-guide/io/arrow.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/io/avro.html b/user-guide/io/avro.html
index 1f33f294..a11b3bef 100644
--- a/user-guide/io/avro.html
+++ b/user-guide/io/avro.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/io/csv.html b/user-guide/io/csv.html
index d99d2f0a..982565f2 100644
--- a/user-guide/io/csv.html
+++ b/user-guide/io/csv.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/io/index.html b/user-guide/io/index.html
index 30b06703..3e4f35d7 100644
--- a/user-guide/io/index.html
+++ b/user-guide/io/index.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/io/json.html b/user-guide/io/json.html
index 63441b69..b4ba740e 100644
--- a/user-guide/io/json.html
+++ b/user-guide/io/json.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/io/parquet.html b/user-guide/io/parquet.html
index 00f0aa18..8880ca6a 100644
--- a/user-guide/io/parquet.html
+++ b/user-guide/io/parquet.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/io/table_provider.html
b/user-guide/io/table_provider.html
index 9558846a..2c45a7ff 100644
--- a/user-guide/io/table_provider.html
+++ b/user-guide/io/table_provider.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/sql.html b/user-guide/sql.html
index e1fe06d1..5604381b 100644
--- a/user-guide/sql.html
+++ b/user-guide/sql.html
@@ -233,6 +233,11 @@
Introduction
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/ffi.html">
+ Python Extensions
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
