This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit c34b2db8f6b45de6c40277a49bfad86c6be7dbcc
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Oct 24 00:45:25 2025 +0000
Publish built docs triggered by 1af3699a29ddaacb7f82ed2e785dd3123f2ca605
---
_sources/contributor-guide/howtos.md.txt | 170 +++++++++++--------
.../functions/adding-udfs.md.txt | 8 +-
contributor-guide/howtos.html | 185 +++++++++++----------
library-user-guide/extending-operators.html | 6 +-
library-user-guide/functions/adding-udfs.html | 16 +-
library-user-guide/functions/index.html | 4 +-
library-user-guide/upgrading.html | 15 +-
searchindex.js | 2 +-
8 files changed, 212 insertions(+), 194 deletions(-)
diff --git a/_sources/contributor-guide/howtos.md.txt
b/_sources/contributor-guide/howtos.md.txt
index 89a1bc7360..24b63865cb 100644
--- a/_sources/contributor-guide/howtos.md.txt
+++ b/_sources/contributor-guide/howtos.md.txt
@@ -21,60 +21,86 @@
## How to update the version of Rust used in CI tests
-- Make a PR to update the [rust-toolchain] file in the root of the repository:
+Make a PR to update the [rust-toolchain] file in the root of the repository.
[rust-toolchain]:
https://github.com/apache/datafusion/blob/main/rust-toolchain.toml
-## How to add a new scalar function
-
-Below is a checklist of what you need to do to add a new scalar function to
DataFusion:
-
-- Add the actual implementation of the function to a new module file within:
- -
[here](https://github.com/apache/datafusion/tree/main/datafusion/functions-nested)
for arrays, maps and structs functions
- -
[here](https://github.com/apache/datafusion/tree/main/datafusion/functions/src/crypto)
for crypto functions
- -
[here](https://github.com/apache/datafusion/tree/main/datafusion/functions/src/datetime)
for datetime functions
- -
[here](https://github.com/apache/datafusion/tree/main/datafusion/functions/src/encoding)
for encoding functions
- -
[here](https://github.com/apache/datafusion/tree/main/datafusion/functions/src/math)
for math functions
- -
[here](https://github.com/apache/datafusion/tree/main/datafusion/functions/src/regex)
for regex functions
- -
[here](https://github.com/apache/datafusion/tree/main/datafusion/functions/src/string)
for string functions
- -
[here](https://github.com/apache/datafusion/tree/main/datafusion/functions/src/unicode)
for unicode functions
- - create a new module
[here](https://github.com/apache/datafusion/tree/main/datafusion/functions/src/)
for other functions.
-- New function modules - for example a `vector` module, should use a [rust
feature](https://doc.rust-lang.org/cargo/reference/features.html) (for example
`vector_expressions`) to allow DataFusion
- users to enable or disable the new module as desired.
-- The implementation of the function is done via implementing `ScalarUDFImpl`
trait for the function struct.
- - See the [advanced_udf.rs] example for an example implementation
- - Add tests for the new function
-- To connect the implementation of the function add to the mod.rs file:
- - a `mod xyz;` where xyz is the new module file
- - a call to `make_udf_function!(..);`
- - an item in `export_functions!(..);`
-- In [sqllogictest/test_files], add new `sqllogictest` integration tests where
the function is called through SQL against well known data and returns the
expected result.
- - Documentation for `sqllogictest`
[here](https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/README.md)
-- Add SQL reference documentation
[here](https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/scalar_functions.md)
- - An example of this being done can be seen
[here](https://github.com/apache/datafusion/pull/12775)
- - Run `./dev/update_function_docs.sh` to update docs
-
-[advanced_udf.rs]:
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_udaf.rs
-[datafusion/expr/src]:
https://github.com/apache/datafusion/tree/main/datafusion/expr/src
-[sqllogictest/test_files]:
https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest/test_files
-
-## How to add a new aggregate function
-
-Below is a checklist of what you need to do to add a new aggregate function to
DataFusion:
-
-- Add the actual implementation of an `Accumulator` and `AggregateExpr`:
-- In [datafusion/expr/src], add:
- - a new variant to `AggregateFunction`
- - a new entry to `FromStr` with the name of the function as called by SQL
- - a new line in `return_type` with the expected return type of the function,
given an incoming type
- - a new line in `signature` with the signature of the function (number and
types of its arguments)
- - a new line in `create_aggregate_expr` mapping the built-in to the
implementation
- - tests to the function.
-- In [sqllogictest/test_files], add new `sqllogictest` integration tests where
the function is called through SQL against well known data and returns the
expected result.
- - Documentation for `sqllogictest`
[here](https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/README.md)
-- Add SQL reference documentation
[here](https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/aggregate_functions.md)
- - An example of this being done can be seen
[here](https://github.com/apache/datafusion/pull/12775)
- - Run `./dev/update_function_docs.sh` to update docs
+## Adding new functions
+
+**Implementation**
+
+| Function type | Location to implement | Trait to implement
| Macros to use | Example
|
+| ------------- | ------------------------- |
---------------------------------------------- |
------------------------------------------------ | -------------------- |
+| Scalar | [functions][df-functions] | [`ScalarUDFImpl`]
| `make_udf_function!()` and `export_functions!()` |
[`advanced_udf.rs`] |
+| Nested | [functions-nested] | [`ScalarUDFImpl`]
| `make_udf_expr_and_func!()` |
|
+| Aggregate | [functions-aggregate] | [`AggregateUDFImpl`] and an
[`Accumulator`] | `make_udaf_expr_and_func!()` |
[`advanced_udaf.rs`] |
+| Window | [functions-window] | [`WindowUDFImpl`] and a
[`PartitionEvaluator`] | `define_udwf_and_expr!()` |
[`advanced_udwf.rs`] |
+| Table | [functions-table] | [`TableFunctionImpl`] and a
[`TableProvider`] | `create_udtf_function!()` |
[`simple_udtf.rs`] |
+
+- The macros are to simplify some boilerplate such as ensuring a DataFrame API
compatible function is also created
+- Ensure new functions are properly exported through the subproject
+ `mod.rs` or `lib.rs`.
+- Functions should preferably provide documentation via the `#[user_doc(...)]`
attribute so their documentation
+ can be included in the SQL reference documentation (see below section)
+- Scalar functions are further grouped into modules for families of functions
(e.g. string, math, datetime).
+ Functions should be added to the relevant module; if a new module needs to
be created then a new [Rust feature]
+ should also be added to allow DataFusion users to conditionally compile the
modules as needed
+- Aggregate functions can optionally implement a [`GroupsAccumulator`] for
better performance
+
+Spark compatible functions are [located in separate crate][df-spark] but
otherwise follow the same steps, though all
+function types (e.g. scalar, nested, aggregate) are grouped together in the
single location.
+
+[df-functions]:
https://github.com/apache/datafusion/tree/main/datafusion/functions
+[functions-nested]:
https://github.com/apache/datafusion/tree/main/datafusion/functions-nested
+[functions-aggregate]:
https://github.com/apache/datafusion/tree/main/datafusion/functions-aggregate
+[functions-window]:
https://github.com/apache/datafusion/tree/main/datafusion/functions-window
+[functions-table]:
https://github.com/apache/datafusion/tree/main/datafusion/functions-table
+[df-spark]: https://github.com/apache/datafusion/tree/main/datafusion/spark
+[`scalarudfimpl`]:
https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html
+[`aggregateudfimpl`]:
https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.AggregateUDFImpl.html
+[`accumulator`]:
https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.Accumulator.html
+[`groupsaccumulator`]:
https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.GroupsAccumulator.html
+[`windowudfimpl`]:
https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.WindowUDFImpl.html
+[`partitionevaluator`]:
https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.PartitionEvaluator.html
+[`tablefunctionimpl`]:
https://docs.rs/datafusion/latest/datafusion/catalog/trait.TableFunctionImpl.html
+[`tableprovider`]:
https://docs.rs/datafusion/latest/datafusion/catalog/trait.TableProvider.html
+[`advanced_udf.rs`]:
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_udf.rs
+[`advanced_udaf.rs`]:
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_udaf.rs
+[`advanced_udwf.rs`]:
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_udwf.rs
+[`simple_udtf.rs`]:
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/simple_udtf.rs
+[rust feature]: https://doc.rust-lang.org/cargo/reference/features.html
+
+**Testing**
+
+Prefer adding `sqllogictest` integration tests where the function is called
via SQL against
+well known data and returns an expected result. See the existing [test
files][slt-test-files] if
+there is an appropriate file to add test cases to, otherwise create a new
file. See the
+[`sqllogictest` documentation][slt-readme] for details on how to construct
these tests.
+Ensure edge case, `null` input cases are considered in these tests.
+
+If a behaviour cannot be tested via `sqllogictest` (e.g. testing `simplify()`,
needs to be
+tested in isolation from the optimizer, difficult to construct exact input via
`sqllogictest`)
+then tests can be added as Rust unit tests in the implementation module,
though these should be
+kept minimal where possible
+
+[slt-test-files]:
https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest/test_files
+[slt-readme]:
https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/README.md
+
+**Documentation**
+
+Run documentation update script `./dev/update_function_docs.sh` which will
update the relevant
+markdown document [here][fn-doc-home] (see the documents for
[scalar][fn-doc-scalar],
+[aggregate][fn-doc-aggregate] and [window][fn-doc-window] functions)
+
+- You _should not_ manually update the markdown document after running the
script as those manual
+ changes would be overwritten on next execution
+- Reference [GitHub issue] which introduced this behaviour
+
+[fn-doc-home]:
https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql
+[fn-doc-scalar]:
https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/scalar_functions.md
+[fn-doc-aggregate]:
https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/aggregate_functions.md
+[fn-doc-window]:
https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/window_functions.md
+[github issue]: https://github.com/apache/datafusion/issues/12740
## How to display plans graphically
@@ -97,11 +123,13 @@ can be displayed. For example, the following command
creates a
dot -Tpdf < /tmp/plan.dot > /tmp/plan.pdf
```
-## How to format `.md` document
+## How to format `.md` documents
-We are using `prettier` to format `.md` files.
+We use [`prettier`] to format `.md` files.
-You can either use `npm i -g prettier` to install it globally or use `npx` to
run it as a standalone binary. Using `npx` required a working node environment.
Upgrading to the latest prettier is recommended (by adding `--upgrade` to the
`npm` command).
+You can either use `npm i -g prettier` to install it globally or use `npx` to
run it as a standalone binary.
+Using `npx` requires a working node environment. Upgrading to the latest
prettier is recommended (by adding
+`--upgrade` to the `npm` command).
```bash
$ prettier --version
@@ -114,19 +142,19 @@ After you've confirmed your prettier version, you can
format all the `.md` files
prettier -w {datafusion,datafusion-cli,datafusion-examples,dev,docs}/**/*.md
```
+[`prettier`]: https://prettier.io/
+
## How to format `.toml` files
-We use `taplo` to format `.toml` files.
+We use [`taplo`] to format `.toml` files.
-For Rust developers, you can install it via:
+To install via cargo:
```sh
cargo install taplo-cli --locked
```
-> Refer to the [Installation section][doc] on other ways to install it.
->
-> [doc]: https://taplo.tamasfe.dev/cli/installation/binary.html
+> Refer to the [taplo installation documentation][taplo-install] for other
ways to install it.
```bash
$ taplo --version
@@ -139,28 +167,24 @@ After you've confirmed your `taplo` version, you can
format all the `.toml` file
taplo fmt
```
+[`taplo`]: https://taplo.tamasfe.dev/
+[taplo-install]: https://taplo.tamasfe.dev/cli/installation/binary.html
+
## How to update protobuf/gen dependencies
-The prost/tonic code can be generated by running `./regen.sh`, which in turn
invokes the Rust binary located in `./gen`
+For the `proto` and `proto-common` crates, the prost/tonic code is generated
by running their respective `./regen.sh` scripts,
+which in turn invokes the Rust binary located in `./gen`.
This is necessary after modifying the protobuf definitions or altering the
dependencies of `./gen`, and requires a
valid installation of [protoc] (see [installation instructions] for details).
```bash
-./regen.sh
+# From repository root
+# proto-common
+./datafusion/proto-common/regen.sh
+# proto
+./datafusion/proto/regen.sh
```
[protoc]:
https://github.com/protocolbuffers/protobuf#protocol-compiler-installation
[installation instructions]:
https://datafusion.apache.org/contributor-guide/getting_started.html#protoc-installation
-
-## How to add/edit documentation for UDFs
-
-Documentations for the UDF documentations are generated from code (related
[github issue]). To generate markdown run `./update_function_docs.sh`.
-
-This is necessary after adding new UDF implementation or modifying existing
implementation which requires to update documentation.
-
-```bash
-./dev/update_function_docs.sh
-```
-
-[github issue]: https://github.com/apache/datafusion/issues/12740
diff --git a/_sources/library-user-guide/functions/adding-udfs.md.txt
b/_sources/library-user-guide/functions/adding-udfs.md.txt
index 2335105882..ecb618179e 100644
--- a/_sources/library-user-guide/functions/adding-udfs.md.txt
+++ b/_sources/library-user-guide/functions/adding-udfs.md.txt
@@ -354,7 +354,7 @@ async fn main() {
}
```
-## Adding a Async Scalar UDF
+## Adding an Async Scalar UDF
An Async Scalar UDF allows you to implement user-defined functions that support
asynchronous execution, such as performing network or I/O operations within the
@@ -1257,7 +1257,7 @@ async fn main() -> Result<()> {
[`create_udaf`]:
https://docs.rs/datafusion/latest/datafusion/logical_expr/fn.create_udaf.html
[`advanced_udaf.rs`]:
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_udaf.rs
-## Adding a User-Defined Table Function
+## Adding a Table UDF
A User-Defined Table Function (UDTF) is a function that takes parameters and
returns a `TableProvider`.
@@ -1266,8 +1266,8 @@ This is a simple struct that holds a set of RecordBatches
in memory and treats t
be replaced with your own struct that implements `TableProvider`.
While this is a simple example for illustrative purposes, UDTFs have a lot of
potential use cases. And can be
-particularly useful for reading data from external sources and interactive
analysis. For example, see the [example][4]
-for a working example that reads from a CSV file. As another example, you
could use the built-in UDTF `parquet_metadata`
+particularly useful for reading data from external sources and interactive
analysis. See the [working example][simple_udtf.rs]
+which reads from a CSV file. As another example, you could use the built-in
UDTF `parquet_metadata`
in the CLI to read the metadata from a Parquet file.
```console
diff --git a/contributor-guide/howtos.html b/contributor-guide/howtos.html
index 4b2eb5ca68..3cccbdf782 100644
--- a/contributor-guide/howtos.html
+++ b/contributor-guide/howtos.html
@@ -718,81 +718,85 @@
<h1>HOWTOs<a class="headerlink" href="#howtos" title="Link to this
heading">#</a></h1>
<section id="how-to-update-the-version-of-rust-used-in-ci-tests">
<h2>How to update the version of Rust used in CI tests<a class="headerlink"
href="#how-to-update-the-version-of-rust-used-in-ci-tests" title="Link to this
heading">#</a></h2>
-<ul class="simple">
-<li><p>Make a PR to update the <a class="reference external"
href="https://github.com/apache/datafusion/blob/main/rust-toolchain.toml";>rust-toolchain</a>
file in the root of the repository:</p></li>
-</ul>
+<p>Make a PR to update the <a class="reference external"
href="https://github.com/apache/datafusion/blob/main/rust-toolchain.toml";>rust-toolchain</a>
file in the root of the repository.</p>
</section>
-<section id="how-to-add-a-new-scalar-function">
-<h2>How to add a new scalar function<a class="headerlink"
href="#how-to-add-a-new-scalar-function" title="Link to this heading">#</a></h2>
-<p>Below is a checklist of what you need to do to add a new scalar function to
DataFusion:</p>
+<section id="adding-new-functions">
+<h2>Adding new functions<a class="headerlink" href="#adding-new-functions"
title="Link to this heading">#</a></h2>
+<p><strong>Implementation</strong></p>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Function type</p></th>
+<th class="head"><p>Location to implement</p></th>
+<th class="head"><p>Trait to implement</p></th>
+<th class="head"><p>Macros to use</p></th>
+<th class="head"><p>Example</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p>Scalar</p></td>
+<td><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions";>functions</a></p></td>
+<td><p><a class="reference external"
href="https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html";><code
class="docutils literal notranslate"><span
class="pre">ScalarUDFImpl</span></code></a></p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">make_udf_function!()</span></code> and <code class="docutils
literal notranslate"><span
class="pre">export_functions!()</span></code></p></td>
+<td><p><a class="reference external"
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_udf.rs";><code
class="docutils literal notranslate"><span
class="pre">advanced_udf.rs</span></code></a></p></td>
+</tr>
+<tr class="row-odd"><td><p>Nested</p></td>
+<td><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions-nested";>functions-nested</a></p></td>
+<td><p><a class="reference external"
href="https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html";><code
class="docutils literal notranslate"><span
class="pre">ScalarUDFImpl</span></code></a></p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">make_udf_expr_and_func!()</span></code></p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-even"><td><p>Aggregate</p></td>
+<td><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions-aggregate";>functions-aggregate</a></p></td>
+<td><p><a class="reference external"
href="https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.AggregateUDFImpl.html";><code
class="docutils literal notranslate"><span
class="pre">AggregateUDFImpl</span></code></a> and an <a class="reference
external"
href="https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.Accumulator.html";><code
class="docutils literal notranslate"><span
class="pre">Accumulator</span></code></a></p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">make_udaf_expr_and_func!()</span></code></p></td>
+<td><p><a class="reference external"
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_udaf.rs";><code
class="docutils literal notranslate"><span
class="pre">advanced_udaf.rs</span></code></a></p></td>
+</tr>
+<tr class="row-odd"><td><p>Window</p></td>
+<td><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions-window";>functions-window</a></p></td>
+<td><p><a class="reference external"
href="https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.WindowUDFImpl.html";><code
class="docutils literal notranslate"><span
class="pre">WindowUDFImpl</span></code></a> and a <a class="reference external"
href="https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.PartitionEvaluator.html";><code
class="docutils literal notranslate"><span
class="pre">PartitionEvaluator</span></code></a></p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">define_udwf_and_expr!()</span></code></p></td>
+<td><p><a class="reference external"
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_udwf.rs";><code
class="docutils literal notranslate"><span
class="pre">advanced_udwf.rs</span></code></a></p></td>
+</tr>
+<tr class="row-even"><td><p>Table</p></td>
+<td><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions-table";>functions-table</a></p></td>
+<td><p><a class="reference external"
href="https://docs.rs/datafusion/latest/datafusion/catalog/trait.TableFunctionImpl.html";><code
class="docutils literal notranslate"><span
class="pre">TableFunctionImpl</span></code></a> and a <a class="reference
external"
href="https://docs.rs/datafusion/latest/datafusion/catalog/trait.TableProvider.html";><code
class="docutils literal notranslate"><span
class="pre">TableProvider</span></code></a></p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">create_udtf_function!()</span></code></p></td>
+<td><p><a class="reference external"
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/simple_udtf.rs";><code
class="docutils literal notranslate"><span
class="pre">simple_udtf.rs</span></code></a></p></td>
+</tr>
+</tbody>
+</table>
+</div>
<ul class="simple">
-<li><p>Add the actual implementation of the function to a new module file
within:</p>
-<ul>
-<li><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions-nested";>here</a>
for arrays, maps and structs functions</p></li>
-<li><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions/src/crypto";>here</a>
for crypto functions</p></li>
-<li><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions/src/datetime";>here</a>
for datetime functions</p></li>
-<li><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions/src/encoding";>here</a>
for encoding functions</p></li>
-<li><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions/src/math";>here</a>
for math functions</p></li>
-<li><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions/src/regex";>here</a>
for regex functions</p></li>
-<li><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions/src/string";>here</a>
for string functions</p></li>
-<li><p><a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions/src/unicode";>here</a>
for unicode functions</p></li>
-<li><p>create a new module <a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/functions/src/";>here</a>
for other functions.</p></li>
-</ul>
-</li>
-<li><p>New function modules - for example a <code class="docutils literal
notranslate"><span class="pre">vector</span></code> module, should use a <a
class="reference external"
href="https://doc.rust-lang.org/cargo/reference/features.html";>rust feature</a>
(for example <code class="docutils literal notranslate"><span
class="pre">vector_expressions</span></code>) to allow DataFusion
-users to enable or disable the new module as desired.</p></li>
-<li><p>The implementation of the function is done via implementing <code
class="docutils literal notranslate"><span
class="pre">ScalarUDFImpl</span></code> trait for the function struct.</p>
-<ul>
-<li><p>See the <a class="reference external"
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_udaf.rs";>advanced_udf.rs</a>
example for an example implementation</p></li>
-<li><p>Add tests for the new function</p></li>
-</ul>
-</li>
-<li><p>To connect the implementation of the function add to the mod.rs
file:</p>
-<ul>
-<li><p>a <code class="docutils literal notranslate"><span
class="pre">mod</span> <span class="pre">xyz;</span></code> where xyz is the
new module file</p></li>
-<li><p>a call to <code class="docutils literal notranslate"><span
class="pre">make_udf_function!(..);</span></code></p></li>
-<li><p>an item in <code class="docutils literal notranslate"><span
class="pre">export_functions!(..);</span></code></p></li>
-</ul>
-</li>
-<li><p>In <a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest/test_files";>sqllogictest/test_files</a>,
add new <code class="docutils literal notranslate"><span
class="pre">sqllogictest</span></code> integration tests where the function is
called through SQL against well known data and returns the expected result.</p>
-<ul>
-<li><p>Documentation for <code class="docutils literal notranslate"><span
class="pre">sqllogictest</span></code> <a class="reference external"
href="https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/README.md";>here</a></p></li>
+<li><p>The macros are to simplify some boilerplate such as ensuring a
DataFrame API compatible function is also created</p></li>
+<li><p>Ensure new functions are properly exported through the subproject
+<code class="docutils literal notranslate"><span
class="pre">mod.rs</span></code> or <code class="docutils literal
notranslate"><span class="pre">lib.rs</span></code>.</p></li>
+<li><p>Functions should preferably provide documentation via the <code
class="docutils literal notranslate"><span
class="pre">#[user_doc(...)]</span></code> attribute so their documentation
+can be included in the SQL reference documentation (see below section)</p></li>
+<li><p>Scalar functions are further grouped into modules for families of
functions (e.g. string, math, datetime).
+Functions should be added to the relevant module; if a new module needs to be
created then a new <a class="reference external"
href="https://doc.rust-lang.org/cargo/reference/features.html";>Rust feature</a>
+should also be added to allow DataFusion users to conditionally compile the
modules as needed</p></li>
+<li><p>Aggregate functions can optionally implement a <a class="reference
external"
href="https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.GroupsAccumulator.html";><code
class="docutils literal notranslate"><span
class="pre">GroupsAccumulator</span></code></a> for better performance</p></li>
</ul>
-</li>
-<li><p>Add SQL reference documentation <a class="reference external"
href="https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/scalar_functions.md";>here</a></p>
-<ul>
-<li><p>An example of this being done can be seen <a class="reference external"
href="https://github.com/apache/datafusion/pull/12775";>here</a></p></li>
-<li><p>Run <code class="docutils literal notranslate"><span
class="pre">./dev/update_function_docs.sh</span></code> to update docs</p></li>
-</ul>
-</li>
-</ul>
-</section>
-<section id="how-to-add-a-new-aggregate-function">
-<h2>How to add a new aggregate function<a class="headerlink"
href="#how-to-add-a-new-aggregate-function" title="Link to this
heading">#</a></h2>
-<p>Below is a checklist of what you need to do to add a new aggregate function
to DataFusion:</p>
+<p>Spark compatible functions are <a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/spark";>located
in separate crate</a> but otherwise follow the same steps, though all
+function types (e.g. scalar, nested, aggregate) are grouped together in the
single location.</p>
+<p><strong>Testing</strong></p>
+<p>Prefer adding <code class="docutils literal notranslate"><span
class="pre">sqllogictest</span></code> integration tests where the function is
called via SQL against
+well known data and returns an expected result. See the existing <a
class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest/test_files";>test
files</a> if
+there is an appropriate file to add test cases to, otherwise create a new
file. See the
+<a class="reference external"
href="https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/README.md";><code
class="docutils literal notranslate"><span
class="pre">sqllogictest</span></code> documentation</a> for details on how to
construct these tests.
+Ensure edge case, <code class="docutils literal notranslate"><span
class="pre">null</span></code> input cases are considered in these tests.</p>
+<p>If a behaviour cannot be tested via <code class="docutils literal
notranslate"><span class="pre">sqllogictest</span></code> (e.g. testing <code
class="docutils literal notranslate"><span
class="pre">simplify()</span></code>, needs to be
+tested in isolation from the optimizer, difficult to construct exact input via
<code class="docutils literal notranslate"><span
class="pre">sqllogictest</span></code>)
+then tests can be added as Rust unit tests in the implementation module,
though these should be
+kept minimal where possible</p>
+<p><strong>Documentation</strong></p>
+<p>Run documentation update script <code class="docutils literal
notranslate"><span class="pre">./dev/update_function_docs.sh</span></code>
which will update the relevant
+markdown document <a class="reference external"
href="https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql";>here</a>
(see the documents for <a class="reference external"
href="https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/scalar_functions.md";>scalar</a>,
+<a class="reference external"
href="https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/aggregate_functions.md";>aggregate</a>
and <a class="reference external"
href="https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/window_functions.md";>window</a>
functions)</p>
<ul class="simple">
-<li><p>Add the actual implementation of an <code class="docutils literal
notranslate"><span class="pre">Accumulator</span></code> and <code
class="docutils literal notranslate"><span
class="pre">AggregateExpr</span></code>:</p></li>
-<li><p>In <a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/expr/src";>datafusion/expr/src</a>,
add:</p>
-<ul>
-<li><p>a new variant to <code class="docutils literal notranslate"><span
class="pre">AggregateFunction</span></code></p></li>
-<li><p>a new entry to <code class="docutils literal notranslate"><span
class="pre">FromStr</span></code> with the name of the function as called by
SQL</p></li>
-<li><p>a new line in <code class="docutils literal notranslate"><span
class="pre">return_type</span></code> with the expected return type of the
function, given an incoming type</p></li>
-<li><p>a new line in <code class="docutils literal notranslate"><span
class="pre">signature</span></code> with the signature of the function (number
and types of its arguments)</p></li>
-<li><p>a new line in <code class="docutils literal notranslate"><span
class="pre">create_aggregate_expr</span></code> mapping the built-in to the
implementation</p></li>
-<li><p>tests to the function.</p></li>
-</ul>
-</li>
-<li><p>In <a class="reference external"
href="https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest/test_files";>sqllogictest/test_files</a>,
add new <code class="docutils literal notranslate"><span
class="pre">sqllogictest</span></code> integration tests where the function is
called through SQL against well known data and returns the expected result.</p>
-<ul>
-<li><p>Documentation for <code class="docutils literal notranslate"><span
class="pre">sqllogictest</span></code> <a class="reference external"
href="https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/README.md";>here</a></p></li>
-</ul>
-</li>
-<li><p>Add SQL reference documentation <a class="reference external"
href="https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/aggregate_functions.md";>here</a></p>
-<ul>
-<li><p>An example of this being done can be seen <a class="reference external"
href="https://github.com/apache/datafusion/pull/12775";>here</a></p></li>
-<li><p>Run <code class="docutils literal notranslate"><span
class="pre">./dev/update_function_docs.sh</span></code> to update docs</p></li>
-</ul>
-</li>
+<li><p>You <em>should not</em> manually update the markdown document after
running the script as those manual
+changes would be overwritten on next execution</p></li>
+<li><p>Reference <a class="reference external"
href="https://github.com/apache/datafusion/issues/12740";>GitHub issue</a> which
introduced this behaviour</p></li>
</ul>
</section>
<section id="how-to-display-plans-graphically">
@@ -812,10 +816,12 @@ can be displayed. For example, the following command
creates a
</pre></div>
</div>
</section>
-<section id="how-to-format-md-document">
-<h2>How to format <code class="docutils literal notranslate"><span
class="pre">.md</span></code> document<a class="headerlink"
href="#how-to-format-md-document" title="Link to this heading">#</a></h2>
-<p>We are using <code class="docutils literal notranslate"><span
class="pre">prettier</span></code> to format <code class="docutils literal
notranslate"><span class="pre">.md</span></code> files.</p>
-<p>You can either use <code class="docutils literal notranslate"><span
class="pre">npm</span> <span class="pre">i</span> <span class="pre">-g</span>
<span class="pre">prettier</span></code> to install it globally or use <code
class="docutils literal notranslate"><span class="pre">npx</span></code> to run
it as a standalone binary. Using <code class="docutils literal
notranslate"><span class="pre">npx</span></code> required a working node
environment. Upgrading to the latest prettier is r [...]
+<section id="how-to-format-md-documents">
+<h2>How to format <code class="docutils literal notranslate"><span
class="pre">.md</span></code> documents<a class="headerlink"
href="#how-to-format-md-documents" title="Link to this heading">#</a></h2>
+<p>We use <a class="reference external" href="https://prettier.io/";><code
class="docutils literal notranslate"><span
class="pre">prettier</span></code></a> to format <code class="docutils literal
notranslate"><span class="pre">.md</span></code> files.</p>
+<p>You can either use <code class="docutils literal notranslate"><span
class="pre">npm</span> <span class="pre">i</span> <span class="pre">-g</span>
<span class="pre">prettier</span></code> to install it globally or use <code
class="docutils literal notranslate"><span class="pre">npx</span></code> to run
it as a standalone binary.
+Using <code class="docutils literal notranslate"><span
class="pre">npx</span></code> requires a working node environment. Upgrading to
the latest prettier is recommended (by adding
+<code class="docutils literal notranslate"><span
class="pre">--upgrade</span></code> to the <code class="docutils literal
notranslate"><span class="pre">npm</span></code> command).</p>
<div class="highlight-bash notranslate"><div
class="highlight"><pre><span></span>$<span class="w"> </span>prettier<span
class="w"> </span>--version
<span class="m">2</span>.3.0
</pre></div>
@@ -827,13 +833,13 @@ can be displayed. For example, the following command
creates a
</section>
<section id="how-to-format-toml-files">
<h2>How to format <code class="docutils literal notranslate"><span
class="pre">.toml</span></code> files<a class="headerlink"
href="#how-to-format-toml-files" title="Link to this heading">#</a></h2>
-<p>We use <code class="docutils literal notranslate"><span
class="pre">taplo</span></code> to format <code class="docutils literal
notranslate"><span class="pre">.toml</span></code> files.</p>
-<p>For Rust developers, you can install it via:</p>
+<p>We use <a class="reference external"
href="https://taplo.tamasfe.dev/";><code class="docutils literal
notranslate"><span class="pre">taplo</span></code></a> to format <code
class="docutils literal notranslate"><span class="pre">.toml</span></code>
files.</p>
+<p>To install via cargo:</p>
<div class="highlight-sh notranslate"><div
class="highlight"><pre><span></span>cargo<span class="w"> </span>install<span
class="w"> </span>taplo-cli<span class="w"> </span>--locked
</pre></div>
</div>
<blockquote>
-<div><p>Refer to the <a class="reference external"
href="https://taplo.tamasfe.dev/cli/installation/binary.html";>Installation
section</a> on other ways to install it.</p>
+<div><p>Refer to the <a class="reference external"
href="https://taplo.tamasfe.dev/cli/installation/binary.html";>taplo
installation documentation</a> for other ways to install it.</p>
</div></blockquote>
<div class="highlight-bash notranslate"><div
class="highlight"><pre><span></span>$<span class="w"> </span>taplo<span
class="w"> </span>--version
taplo<span class="w"> </span><span class="m">0</span>.9.0
@@ -846,18 +852,15 @@ taplo<span class="w"> </span><span class="m">0</span>.9.0
</section>
<section id="how-to-update-protobuf-gen-dependencies">
<h2>How to update protobuf/gen dependencies<a class="headerlink"
href="#how-to-update-protobuf-gen-dependencies" title="Link to this
heading">#</a></h2>
-<p>The prost/tonic code can be generated by running <code class="docutils
literal notranslate"><span class="pre">./regen.sh</span></code>, which in turn
invokes the Rust binary located in <code class="docutils literal
notranslate"><span class="pre">./gen</span></code></p>
+<p>For the <code class="docutils literal notranslate"><span
class="pre">proto</span></code> and <code class="docutils literal
notranslate"><span class="pre">proto-common</span></code> crates, the
prost/tonic code is generated by running their respective <code class="docutils
literal notranslate"><span class="pre">./regen.sh</span></code> scripts,
+which in turn invokes the Rust binary located in <code class="docutils literal
notranslate"><span class="pre">./gen</span></code>.</p>
<p>This is necessary after modifying the protobuf definitions or altering the
dependencies of <code class="docutils literal notranslate"><span
class="pre">./gen</span></code>, and requires a
valid installation of <a class="reference external"
href="https://github.com/protocolbuffers/protobuf#protocol-compiler-installation";>protoc</a>
(see <a class="reference external"
href="https://datafusion.apache.org/contributor-guide/getting_started.html#protoc-installation";>installation
instructions</a> for details).</p>
-<div class="highlight-bash notranslate"><div
class="highlight"><pre><span></span>./regen.sh
-</pre></div>
-</div>
-</section>
-<section id="how-to-add-edit-documentation-for-udfs">
-<h2>How to add/edit documentation for UDFs<a class="headerlink"
href="#how-to-add-edit-documentation-for-udfs" title="Link to this
heading">#</a></h2>
-<p>Documentations for the UDF documentations are generated from code (related
<a class="reference external"
href="https://github.com/apache/datafusion/issues/12740";>github issue</a>). To
generate markdown run <code class="docutils literal notranslate"><span
class="pre">./update_function_docs.sh</span></code>.</p>
-<p>This is necessary after adding new UDF implementation or modifying existing
implementation which requires to update documentation.</p>
-<div class="highlight-bash notranslate"><div
class="highlight"><pre><span></span>./dev/update_function_docs.sh
+<div class="highlight-bash notranslate"><div
class="highlight"><pre><span></span><span class="c1"># From repository
root</span>
+<span class="c1"># proto-common</span>
+./datafusion/proto-common/regen.sh
+<span class="c1"># proto</span>
+./datafusion/proto/regen.sh
</pre></div>
</div>
</section>
@@ -911,13 +914,11 @@ valid installation of <a class="reference external"
href="https://github.com/pro
<nav class="bd-toc-nav page-toc"
aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#how-to-update-the-version-of-rust-used-in-ci-tests">How to update the
version of Rust used in CI tests</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#how-to-add-a-new-scalar-function">How to add a new scalar
function</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#how-to-add-a-new-aggregate-function">How to add a new aggregate
function</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#adding-new-functions">Adding new functions</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#how-to-display-plans-graphically">How to display plans
graphically</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#how-to-format-md-document">How to format <code class="docutils literal
notranslate"><span class="pre">.md</span></code> document</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#how-to-format-md-documents">How to format <code class="docutils literal
notranslate"><span class="pre">.md</span></code> documents</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#how-to-format-toml-files">How to format <code class="docutils literal
notranslate"><span class="pre">.toml</span></code> files</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#how-to-update-protobuf-gen-dependencies">How to update protobuf/gen
dependencies</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#how-to-add-edit-documentation-for-udfs">How to add/edit documentation
for UDFs</a></li>
</ul>
</nav></div>
diff --git a/library-user-guide/extending-operators.html
b/library-user-guide/extending-operators.html
index 2216d0c48a..2402dd8970 100644
--- a/library-user-guide/extending-operators.html
+++ b/library-user-guide/extending-operators.html
@@ -723,8 +723,7 @@
<section id="optimizing-logical-plan">
<h3>Optimizing Logical Plan<a class="headerlink"
href="#optimizing-logical-plan" title="Link to this heading">#</a></h3>
<p>The <code class="docutils literal notranslate"><span
class="pre">rewrite</span></code> function transforms logical plans by
identifying temporal patterns and aggregation functions that match the stored
wheel indices. When match is found, it queries the corresponding index to
retrieve pre-computed aggregate values, stores these results in a <a
class="reference external"
href="https://docs.rs/datafusion/latest/datafusion/datasource/memory/struct.MemTable.html";>MemTable</a>,
and returns [...]
-<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="p">,</span><span
class="n">ignore</span>
-<span class="k">fn</span><span class="w"> </span><span
class="nf">rewrite</span><span class="p">(</span>
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="k">fn</span><span class="w">
</span><span class="nf">rewrite</span><span class="p">(</span>
<span class="w"> </span><span class="o">&</span><span
class="bp">self</span><span class="p">,</span>
<span class="w"> </span><span class="n">plan</span><span
class="p">:</span><span class="w"> </span><span
class="nc">LogicalPlan</span><span class="p">,</span>
<span class="w"> </span><span class="n">_config</span><span
class="p">:</span><span class="w"> </span><span class="kp">&</span><span
class="nc">dyn</span><span class="w"> </span><span
class="n">OptimizerConfig</span><span class="p">,</span>
@@ -739,8 +738,7 @@
<span class="p">}</span>
</pre></div>
</div>
-<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="p">,</span><span
class="n">ignore</span>
-<span class="c1">// Converts a uwheel aggregate result to a TableScan with a
MemTable as source</span>
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="c1">// Converts a uwheel
aggregate result to a TableScan with a MemTable as source</span>
<span class="k">fn</span><span class="w"> </span><span
class="nf">agg_to_table_scan</span><span class="p">(</span><span
class="n">result</span><span class="p">:</span><span class="w"> </span><span
class="kt">f64</span><span class="p">,</span><span class="w"> </span><span
class="n">schema</span><span class="p">:</span><span class="w"> </span><span
class="nc">SchemaRef</span><span class="p">)</span><span class="w">
</span><span class="p">-></span><span class="w"> </span><span class="nb"
[...]
<span class="w"> </span><span class="kd">let</span><span class="w">
</span><span class="n">data</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="n">Float64Array</span><span class="p">::</span><span
class="n">from</span><span class="p">(</span><span class="fm">vec!</span><span
class="p">[</span><span class="n">result</span><span class="p">]);</span>
<span class="w"> </span><span class="kd">let</span><span class="w">
</span><span class="n">record_batch</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="n">RecordBatch</span><span class="p">::</span><span
class="n">try_new</span><span class="p">(</span><span
class="n">schema</span><span class="p">.</span><span
class="n">clone</span><span class="p">(),</span><span class="w"> </span><span
class="fm">vec!</span><span class="p">[</span><span class=" [...]
diff --git a/library-user-guide/functions/adding-udfs.html
b/library-user-guide/functions/adding-udfs.html
index b9390d08ef..9ea6ad1c66 100644
--- a/library-user-guide/functions/adding-udfs.html
+++ b/library-user-guide/functions/adding-udfs.html
@@ -942,8 +942,8 @@ for the same input.</p></li>
</section>
</section>
</section>
-<section id="adding-a-async-scalar-udf">
-<h2>Adding a Async Scalar UDF<a class="headerlink"
href="#adding-a-async-scalar-udf" title="Link to this heading">#</a></h2>
+<section id="adding-an-async-scalar-udf">
+<h2>Adding an Async Scalar UDF<a class="headerlink"
href="#adding-an-async-scalar-udf" title="Link to this heading">#</a></h2>
<p>An Async Scalar UDF allows you to implement user-defined functions that
support
asynchronous execution, such as performing network or I/O operations within the
UDF.</p>
@@ -1404,15 +1404,15 @@ for the same input.</p></li>
</div>
</section>
</section>
-<section id="adding-a-user-defined-table-function">
-<h2>Adding a User-Defined Table Function<a class="headerlink"
href="#adding-a-user-defined-table-function" title="Link to this
heading">#</a></h2>
+<section id="adding-a-table-udf">
+<h2>Adding a Table UDF<a class="headerlink" href="#adding-a-table-udf"
title="Link to this heading">#</a></h2>
<p>A User-Defined Table Function (UDTF) is a function that takes parameters
and returns a <code class="docutils literal notranslate"><span
class="pre">TableProvider</span></code>.</p>
<p>Because we’re returning a <code class="docutils literal notranslate"><span
class="pre">TableProvider</span></code>, in this example we’ll use the <code
class="docutils literal notranslate"><span class="pre">MemTable</span></code>
data source to represent a table.
This is a simple struct that holds a set of RecordBatches in memory and treats
them as a table. In your case, this would
be replaced with your own struct that implements <code class="docutils literal
notranslate"><span class="pre">TableProvider</span></code>.</p>
<p>While this is a simple example for illustrative purposes, UDTFs have a lot
of potential use cases. And can be
-particularly useful for reading data from external sources and interactive
analysis. For example, see the [example][4]
-for a working example that reads from a CSV file. As another example, you
could use the built-in UDTF <code class="docutils literal notranslate"><span
class="pre">parquet_metadata</span></code>
+particularly useful for reading data from external sources and interactive
analysis. See the <a class="reference external"
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/simple_udtf.rs";>working
example</a>
+which reads from a CSV file. As another example, you could use the built-in
UDTF <code class="docutils literal notranslate"><span
class="pre">parquet_metadata</span></code>
in the CLI to read the metadata from a Parquet file.</p>
<div class="highlight-console notranslate"><div
class="highlight"><pre><span></span><span class="go">> select filename,
row_group_id, row_group_num_rows, row_group_bytes, stats_min, stats_max from
parquet_metadata('./benchmarks/data/hits.parquet') where column_id =
17 limit 10;</span>
<span
class="go">+--------------------------------+--------------+--------------------+-----------------+-----------+-----------+</span>
@@ -1626,7 +1626,7 @@ validation of the input <code class="docutils literal
notranslate"><span class="
</li>
</ul>
</li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#adding-a-async-scalar-udf">Adding a Async Scalar UDF</a><ul class="nav
section-nav flex-column">
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#adding-an-async-scalar-udf">Adding an Async Scalar UDF</a><ul class="nav
section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#adding-by-impl-asyncscalarudfimpl">Adding by <code class="docutils
literal notranslate"><span class="pre">impl</span> <span
class="pre">AsyncScalarUDFImpl</span></code></a></li>
</ul>
</li>
@@ -1638,7 +1638,7 @@ validation of the input <code class="docutils literal
notranslate"><span class="
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#registering-an-aggregate-udf">Registering an Aggregate UDF</a></li>
</ul>
</li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#adding-a-user-defined-table-function">Adding a User-Defined Table
Function</a><ul class="nav section-nav flex-column">
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#adding-a-table-udf">Adding a Table UDF</a><ul class="nav section-nav
flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#writing-the-udtf">Writing the UDTF</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#registering-and-using-the-udtf">Registering and Using the UDTF</a></li>
</ul>
diff --git a/library-user-guide/functions/index.html
b/library-user-guide/functions/index.html
index 042be06629..bd1d69736d 100644
--- a/library-user-guide/functions/index.html
+++ b/library-user-guide/functions/index.html
@@ -702,10 +702,10 @@
<ul>
<li class="toctree-l1"><a class="reference internal"
href="adding-udfs.html">Adding User Defined Functions:
Scalar/Window/Aggregate/Table Functions</a><ul>
<li class="toctree-l2"><a class="reference internal"
href="adding-udfs.html#adding-a-scalar-udf">Adding a Scalar UDF</a></li>
-<li class="toctree-l2"><a class="reference internal"
href="adding-udfs.html#adding-a-async-scalar-udf">Adding a Async Scalar
UDF</a></li>
+<li class="toctree-l2"><a class="reference internal"
href="adding-udfs.html#adding-an-async-scalar-udf">Adding an Async Scalar
UDF</a></li>
<li class="toctree-l2"><a class="reference internal"
href="adding-udfs.html#adding-a-window-udf">Adding a Window UDF</a></li>
<li class="toctree-l2"><a class="reference internal"
href="adding-udfs.html#adding-an-aggregate-udf">Adding an Aggregate UDF</a></li>
-<li class="toctree-l2"><a class="reference internal"
href="adding-udfs.html#adding-a-user-defined-table-function">Adding a
User-Defined Table Function</a></li>
+<li class="toctree-l2"><a class="reference internal"
href="adding-udfs.html#adding-a-table-udf">Adding a Table UDF</a></li>
<li class="toctree-l2"><a class="reference internal"
href="adding-udfs.html#custom-expression-planning">Custom Expression
Planning</a></li>
</ul>
</li>
diff --git a/library-user-guide/upgrading.html
b/library-user-guide/upgrading.html
index 81991113fb..f8be902a81 100644
--- a/library-user-guide/upgrading.html
+++ b/library-user-guide/upgrading.html
@@ -864,13 +864,11 @@ returns a <code class="docutils literal
notranslate"><span class="pre">ColumnarV
<h3><code class="docutils literal notranslate"><span
class="pre">ProjectionExpr</span></code> changed from type alias to struct<a
class="headerlink" href="#projectionexpr-changed-from-type-alias-to-struct"
title="Link to this heading">#</a></h3>
<p><code class="docutils literal notranslate"><span
class="pre">ProjectionExpr</span></code> has been changed from a type alias to
a struct with named fields to improve code clarity and maintainability.</p>
<p><strong>Before:</strong></p>
-<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="p">,</span><span
class="n">ignore</span>
-<span class="k">pub</span><span class="w"> </span><span
class="k">type</span><span class="w"> </span><span
class="nc">ProjectionExpr</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="p">(</span><span
class="n">Arc</span><span class="o"><</span><span class="k">dyn</span><span
class="w"> </span><span class="n">PhysicalExpr</span><span
class="o">></span><span class="p">,</span><span class="w"> </span><span
class="nb">String</span><span class="p [...]
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="k">pub</span><span class="w">
</span><span class="k">type</span><span class="w"> </span><span
class="nc">ProjectionExpr</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="p">(</span><span
class="n">Arc</span><span class="o"><</span><span class="k">dyn</span><span
class="w"> </span><span class="n">PhysicalExpr</span><span
class="o">></span><span class [...]
</pre></div>
</div>
<p><strong>After:</strong></p>
-<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="p">,</span><span
class="n">ignore</span>
-<span class="cp">#[derive(Debug, Clone)]</span>
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="cp">#[derive(Debug,
Clone)]</span>
<span class="k">pub</span><span class="w"> </span><span
class="k">struct</span><span class="w"> </span><span
class="nc">ProjectionExpr</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">pub</span><span class="w">
</span><span class="n">expr</span><span class="p">:</span><span class="w">
</span><span class="nc">Arc</span><span class="o"><</span><span
class="k">dyn</span><span class="w"> </span><span
class="n">PhysicalExpr</span><span class="o">></span><span class="p">,</span>
<span class="w"> </span><span class="k">pub</span><span class="w">
</span><span class="n">alias</span><span class="p">:</span><span class="w">
</span><span class="nb">String</span><span class="p">,</span>
@@ -997,13 +995,11 @@ Reimplementation for any custom <code class="docutils
literal notranslate"><span
<h3><code class="docutils literal notranslate"><span
class="pre">FileOpenFuture</span></code> now uses <code class="docutils literal
notranslate"><span class="pre">DataFusionError</span></code> instead of <code
class="docutils literal notranslate"><span
class="pre">ArrowError</span></code><a class="headerlink"
href="#fileopenfuture-now-uses-datafusionerror-instead-of-arrowerror"
title="Link to this heading">#</a></h3>
<p>The <code class="docutils literal notranslate"><span
class="pre">FileOpenFuture</span></code> type alias has been updated to use
<code class="docutils literal notranslate"><span
class="pre">DataFusionError</span></code> instead of <code class="docutils
literal notranslate"><span class="pre">ArrowError</span></code> for its error
type. This change affects the <code class="docutils literal notranslate"><span
class="pre">FileOpener</span></code> trait and any implementations that work wi
[...]
<p><strong>Before:</strong></p>
-<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="p">,</span><span
class="n">ignore</span>
-<span class="k">pub</span><span class="w"> </span><span
class="k">type</span><span class="w"> </span><span
class="nc">FileOpenFuture</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="n">BoxFuture</span><span
class="o"><'</span><span class="nb">static</span><span
class="p">,</span><span class="w"> </span><span class="nb">Result</span><span
class="o"><</span><span class="n">BoxStream</span><span
class="o"><'</span><span class="nb">s [...]
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="k">pub</span><span class="w">
</span><span class="k">type</span><span class="w"> </span><span
class="nc">FileOpenFuture</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="n">BoxFuture</span><span
class="o"><'</span><span class="nb">static</span><span
class="p">,</span><span class="w"> </span><span class="nb">Result</span><span
class="o"><</span>< [...]
</pre></div>
</div>
<p><strong>After:</strong></p>
-<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="p">,</span><span
class="n">ignore</span>
-<span class="k">pub</span><span class="w"> </span><span
class="k">type</span><span class="w"> </span><span
class="nc">FileOpenFuture</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="n">BoxFuture</span><span
class="o"><'</span><span class="nb">static</span><span
class="p">,</span><span class="w"> </span><span class="nb">Result</span><span
class="o"><</span><span class="n">BoxStream</span><span
class="o"><'</span><span class="nb">s [...]
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="k">pub</span><span class="w">
</span><span class="k">type</span><span class="w"> </span><span
class="nc">FileOpenFuture</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="n">BoxFuture</span><span
class="o"><'</span><span class="nb">static</span><span
class="p">,</span><span class="w"> </span><span class="nb">Result</span><span
class="o"><</span>< [...]
</pre></div>
</div>
<p>If you have custom implementations of <code class="docutils literal
notranslate"><span class="pre">FileOpener</span></code> or work directly with
<code class="docutils literal notranslate"><span
class="pre">FileOpenFuture</span></code>, you’ll need to update your error
handling to use <code class="docutils literal notranslate"><span
class="pre">DataFusionError</span></code> instead of <code class="docutils
literal notranslate"><span class="pre">ArrowError</span></code>. The <code clas
[...]
@@ -1024,8 +1020,7 @@ this interface so that these libraries can be used across
different DataFusion v
<section id="added-physicalexpr-is-volatile-node">
<h3>Added <code class="docutils literal notranslate"><span
class="pre">PhysicalExpr::is_volatile_node</span></code><a class="headerlink"
href="#added-physicalexpr-is-volatile-node" title="Link to this
heading">#</a></h3>
<p>We added a method to <code class="docutils literal notranslate"><span
class="pre">PhysicalExpr</span></code> to mark a <code class="docutils literal
notranslate"><span class="pre">PhysicalExpr</span></code> as volatile:</p>
-<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="p">,</span><span
class="n">ignore</span>
-<span class="k">impl</span><span class="w"> </span><span
class="n">PhysicalExpr</span><span class="w"> </span><span
class="k">for</span><span class="w"> </span><span
class="n">MyRandomExpr</span><span class="w"> </span><span class="p">{</span>
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="k">impl</span><span class="w">
</span><span class="n">PhysicalExpr</span><span class="w"> </span><span
class="k">for</span><span class="w"> </span><span
class="n">MyRandomExpr</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">fn</span><span class="w">
</span><span class="nf">is_volatile_node</span><span class="p">(</span><span
class="o">&</span><span class="bp">self</span><span class="p">)</span><span
class="w"> </span><span class="p">-></span><span class="w"> </span><span
class="kt">bool</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="p">}</span>
diff --git a/searchindex.js b/searchindex.js
index fe2df49ea4..2a1cc305e9 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"!=":[[59,"op-neq"]],"!~":[[59,"op-re-not-match"]],"!~*":[[59,"op-re-not-match-i"]],"!~~":[[59,"id19"]],"!~~*":[[59,"id20"]],"#":[[59,"op-bit-xor"]],"%":[[59,"op-modulo"]],"&":[[59,"op-bit-and"]],"(relation,
name) tuples in logical fields and logical columns are
unique":[[13,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[59,"op-multiply"]],"+":[[59,"op-plus"]],"-":[[59,"op-minus"]],"/":[[59,"op-divide"]],"<":[[59,"op-lt"]],"<
[...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"!=":[[59,"op-neq"]],"!~":[[59,"op-re-not-match"]],"!~*":[[59,"op-re-not-match-i"]],"!~~":[[59,"id19"]],"!~~*":[[59,"id20"]],"#":[[59,"op-bit-xor"]],"%":[[59,"op-modulo"]],"&":[[59,"op-bit-and"]],"(relation,
name) tuples in logical fields and logical columns are
unique":[[13,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[59,"op-multiply"]],"+":[[59,"op-plus"]],"-":[[59,"op-minus"]],"/":[[59,"op-divide"]],"<":[[59,"op-lt"]],"<
[...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
