This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 376676c7ab docs: show implementation kind for each expression (#5028)
376676c7ab is described below
commit 376676c7abdd6588c5aec640fa06c8f251fbabb6
Author: Andy Grove <[email protected]>
AuthorDate: Fri Jul 24 16:56:30 2026 -0600
docs: show implementation kind for each expression (#5028)
* docs: show implementation kind for each expression in supported
expressions page
Adds an Implementation column (Native / Codegen dispatch / Hybrid / --)
to every table in the supported expressions user-guide page. Values are
auto-generated by GenerateDocs from the QueryPlanSerde maps: the serde's
trait mixins (NativeOptInAvailable, CodegenDispatchFallback, plain
CometCodegenDispatch, plain CometExpressionSerde) decide the kind, and
Spark's FunctionRegistry bridges SQL function names to Catalyst classes.
Rows for functions that are planned, aliased/rewritten before Comet sees
them, or handled at the operator level fall through to the "--" bucket
and are explained in the notes column.
* docs: document how to regenerate the Implementation column
Update the contributor guide to explain what fills the new Implementation
column in expressions.md and how to regenerate it locally with the
generate-docs Maven profile. Also mention the column in the release-docs
step so it is refreshed alongside the compatibility pages.
---
.../contributor-guide/adding_a_new_expression.md | 21 +
docs/source/contributor-guide/release_process.md | 3 +-
docs/source/user-guide/latest/expressions.md | 981 +++++++++++----------
.../main/scala/org/apache/comet/GenerateDocs.scala | 124 ++-
4 files changed, 643 insertions(+), 486 deletions(-)
diff --git a/docs/source/contributor-guide/adding_a_new_expression.md
b/docs/source/contributor-guide/adding_a_new_expression.md
index dcaf188b55..06454a19b6 100644
--- a/docs/source/contributor-guide/adding_a_new_expression.md
+++ b/docs/source/contributor-guide/adding_a_new_expression.md
@@ -258,6 +258,8 @@ In addition to `getSupportLevel`, which governs runtime
planning decisions, the
These methods do not affect runtime behavior. They are called by
`GenerateDocs` (`spark/src/main/scala/org/apache/comet/GenerateDocs.scala`)
when building the user-facing Compatibility Guide pages. The Markdown templates
live in
`docs/source/user-guide/latest/compatibility/expressions/_category_template/`
(for example, `math.md`, `datetime.md`, `array.md`, `aggregate.md`,
`struct.md`); `docs/build.sh` copies them into per-Spark-version subdirectories
(`spark-3.4/`, `spark-3.5/`, `spark-4 [...]
+`GenerateDocs` also rewrites the **Implementation** column of every table in
[`docs/source/user-guide/latest/expressions.md`](../user-guide/latest/expressions.md)
based on the serde's trait mixins: `NativeOptInAvailable` (or its subtype
`CodegenDispatchFallback`) is reported as `Hybrid`, a plain
`CometCodegenDispatch` is reported as `Codegen dispatch`, and anything else is
reported as `Native`. Rows whose function name is not registered in Spark's
`FunctionRegistry` (or has no matching s [...]
+
Key differences from `getSupportLevel`:
- **No expression instance.** Both methods take no arguments, so they describe
the expression in general rather than a specific call site. Use
`getSupportLevel` for checks that depend on data types, argument values, or
other per-instance details.
@@ -308,6 +310,25 @@ override def getUnsupportedReasons(): Seq[String] = Seq(
.mkString("\n - ", "\n - ", ""))
```
+#### Updating the Supported Expressions Page
+
+The user-facing [Spark Expression
Support](../user-guide/latest/expressions.md) page
(`docs/source/user-guide/latest/expressions.md`) lists every Spark built-in
with its status, implementation kind, and any notes. Unlike the compatibility
pages, this file is committed to the repo, so you need to add a row for your
new expression and commit it.
+
+Add the row to the correct category table (matching the section header, for
example `## math_funcs`). Fill in the Function, Status, and Notes cells and
leave the Implementation cell as `—`; the value is regenerated from the serde
and does not need to be set by hand:
+
+```markdown
+| `myfunc` | ✅ | — | Some optional note |
+```
+
+To regenerate the Implementation column locally, run the `generate-docs`
profile against any supported Spark version:
+
+```shell
+./mvnw -q -pl spark -am -Pspark-3.5 -Pgenerate-docs -DskipTests package \
+ -Dexec.arguments="$(pwd)/docs/source/user-guide/latest/,spark-3.5"
+```
+
+That run rewrites every Implementation cell in place, mapping SQL function
names to Catalyst classes via Spark's `FunctionRegistry` and looking each class
up in `QueryPlanSerde`. Commit the updated `expressions.md` alongside your
serde change. If you forget, a reviewer or a subsequent doc regeneration will
fill it in — an `—` in the column for a supported expression usually means the
row is aliased or rewritten before Comet sees it (for example `to_date` →
`Cast`), not that the row is broken.
+
#### Adding Spark-side Tests for the New Expression
It is important to verify that the new expression is correctly recognized by
the native execution engine and matches the expected Spark behavior. The
preferred way to add test coverage is to write a Comet SQL Test. This approach
is simpler than writing Comet Scala Tests and makes it easy to cover many input
combinations and edge cases.
diff --git a/docs/source/contributor-guide/release_process.md
b/docs/source/contributor-guide/release_process.md
index 63bd77ef22..8de85a6b63 100644
--- a/docs/source/contributor-guide/release_process.md
+++ b/docs/source/contributor-guide/release_process.md
@@ -128,7 +128,8 @@ All release branches stay protected, including older ones,
so released code cann
The docs on `main` contain only template markers; CI fills them at publish
time. A release branch instead
commits the generated content so the archived docs for the release render real
tables. Run the script to
-generate the config reference and the per-Spark-version compatibility pages
and freeze them onto the branch:
+generate the config reference, refresh the Implementation column of the Spark
Expression Support page, and
+produce the per-Spark-version compatibility pages, freezing them onto the
branch:
```shell
./dev/generate-release-docs.sh
diff --git a/docs/source/user-guide/latest/expressions.md
b/docs/source/user-guide/latest/expressions.md
index efbd33ba1c..c292700dbc 100644
--- a/docs/source/user-guide/latest/expressions.md
+++ b/docs/source/user-guide/latest/expressions.md
@@ -46,6 +46,19 @@ Most expressions can also be disabled with
`spark.comet.expression.EXPRNAME.enab
| ✅ Supported | Comet produces Spark-compatible results by default. Some
inputs or forms may fall back to Spark, and any incompatible behavior is opt-in
(off by default). |
| 🔜 Planned | Intended; tracked by an open issue or pull request. |
+## Implementation legend
+
+The **Implementation** column records how Comet executes each expression when
it is not falling back to Spark:
+
+| Implementation | Meaning |
+| --- | --- |
+| Native | Comet's Rust engine evaluates the expression end to end. |
+| Codegen dispatch | Spark's own generated JVM code is evaluated inside the
Comet pipeline. Used when a byte-exact match to Spark matters more than the
native speedup, or when no native path exists. |
+| Hybrid | Both paths exist. Comet picks between them based on input, and the
user can override with `spark.comet.expression.EXPRNAME.allowIncompatible=true`
— see [Native and codegen-dispatch
implementations](compatibility/index.md#native-and-codegen-dispatch-implementations).
|
+| — | No direct wire-up: the row is either planned, is rewritten to another
expression before Comet sees it (for example `to_date` → `Cast`), or is handled
at the operator level rather than the expression level (for example `explode`,
window functions). |
+
+The Implementation column is auto-generated from the serde definitions in
`QueryPlanSerde`; do not edit it by hand.
+
## Not currently planned
Comet focuses acceleration on mainstream relational, string, datetime, math,
and collection
@@ -66,147 +79,147 @@ The tables below list every Spark built-in expression
with its current status.
## agg_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `any` | ✅ | |
-| `any_value` | ✅ | |
-| `approx_count_distinct` | ✅ | |
-| `approx_percentile` | ✅ | Byte, short, int, long, float, and double input;
other input types fall back to Spark |
-| `array_agg` | ✅ | Alias for `collect_list` |
-| `avg` | ✅ | Interval types fall back |
-| `bit_and` | ✅ | |
-| `bit_or` | ✅ | |
-| `bit_xor` | ✅ | |
-| `bool_and` | ✅ | |
-| `bool_or` | ✅ | |
-| `collect_list` | ✅ | |
-| `collect_set` | ✅ | |
-| `corr` | ✅ | |
-| `count` | ✅ | |
-| `count_if` | ✅ | |
-| `covar_pop` | ✅ | |
-| `covar_samp` | ✅ | |
-| `every` | ✅ | |
-| `first` | ✅ | |
-| `first_value` | ✅ | |
-| `grouping` | ✅ | Grouping indicator for ROLLUP/CUBE/GROUPING SETS |
-| `grouping_id` | ✅ | Grouping indicator for ROLLUP/CUBE/GROUPING SETS |
-| `kurtosis` | 🔜 | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
-| `last` | ✅ | |
-| `last_value` | ✅ | |
-| `listagg` | 🔜 | String aggregation |
-| `max` | ✅ | |
-| `max_by` | 🔜 |
[#3841](https://github.com/apache/datafusion-comet/issues/3841) |
-| `mean` | ✅ | |
-| `median` | ✅ | Rewrites to `percentile(col, 0.5)` and runs natively for
supported percentile inputs |
-| `min` | ✅ | |
-| `min_by` | 🔜 |
[#3841](https://github.com/apache/datafusion-comet/issues/3841) |
-| `mode` | 🔜 | [#3970](https://github.com/apache/datafusion-comet/issues/3970)
|
-| `percentile` | ✅ | Single literal percentage on numeric input runs natively;
array of percentages and a frequency argument fall back to Spark |
-| `percentile_cont` | ✅ | Spark 4.0+ `WITHIN GROUP (ORDER BY ...)`; ascending
only runs natively, `DESC` falls back to Spark |
-| `percentile_disc` | 🔜 | Percentile aggregate |
-| `regr_avgx` | ✅ | Native: Spark rewrites to `Average` (tests in
[#4551](https://github.com/apache/datafusion-comet/issues/4551)) |
-| `regr_avgy` | ✅ | Native: Spark rewrites to `Average` (tests in
[#4551](https://github.com/apache/datafusion-comet/issues/4551)) |
-| `regr_count` | ✅ | Native: Spark rewrites to `Count` (tests in
[#4551](https://github.com/apache/datafusion-comet/issues/4551)) |
-| `regr_intercept` | 🔜 | Falls back; can reuse `covar_pop`/`var_pop`
accumulators ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
-| `regr_r2` | 🔜 | Falls back; can reuse the `corr` accumulator
([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
-| `regr_slope` | 🔜 | Falls back; can reuse `covar_pop`/`var_pop` accumulators
([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
-| `regr_sxx` | 🔜 | Falls back; can reuse `var_pop` accumulator
([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
-| `regr_sxy` | 🔜 | Falls back; can reuse `covar_pop` accumulator
([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
-| `regr_syy` | 🔜 | Falls back; can reuse `var_pop` accumulator
([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
-| `skewness` | 🔜 | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
-| `some` | ✅ | |
-| `std` | ✅ | |
-| `stddev` | ✅ | |
-| `stddev_pop` | ✅ | |
-| `stddev_samp` | ✅ | |
-| `string_agg` | 🔜 | String aggregation (alias of `listagg`) |
-| `sum` | ✅ | |
-| `try_avg` | ✅ | Interval types fall back |
-| `try_sum` | ✅ | |
-| `var_pop` | ✅ | |
-| `var_samp` | ✅ | |
-| `variance` | ✅ | |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `any` | ✅ | — | |
+| `any_value` | ✅ | — | |
+| `approx_count_distinct` | ✅ | Native | |
+| `approx_percentile` | ✅ | Native | Byte, short, int, long, float, and double
input; other input types fall back to Spark |
+| `array_agg` | ✅ | Native | Alias for `collect_list` |
+| `avg` | ✅ | Native | Interval types fall back |
+| `bit_and` | ✅ | Native | |
+| `bit_or` | ✅ | Native | |
+| `bit_xor` | ✅ | Native | |
+| `bool_and` | ✅ | — | |
+| `bool_or` | ✅ | — | |
+| `collect_list` | ✅ | Native | |
+| `collect_set` | ✅ | Native | |
+| `corr` | ✅ | Native | |
+| `count` | ✅ | Native | |
+| `count_if` | ✅ | — | |
+| `covar_pop` | ✅ | Native | |
+| `covar_samp` | ✅ | Native | |
+| `every` | ✅ | — | |
+| `first` | ✅ | Native | |
+| `first_value` | ✅ | Native | |
+| `grouping` | ✅ | — | Grouping indicator for ROLLUP/CUBE/GROUPING SETS |
+| `grouping_id` | ✅ | — | Grouping indicator for ROLLUP/CUBE/GROUPING SETS |
+| `kurtosis` | 🔜 | — | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `last` | ✅ | Native | |
+| `last_value` | ✅ | Native | |
+| `listagg` | 🔜 | — | String aggregation |
+| `max` | ✅ | Native | |
+| `max_by` | 🔜 | — |
[#3841](https://github.com/apache/datafusion-comet/issues/3841) |
+| `mean` | ✅ | Native | |
+| `median` | ✅ | — | Rewrites to `percentile(col, 0.5)` and runs natively for
supported percentile inputs |
+| `min` | ✅ | Native | |
+| `min_by` | 🔜 | — |
[#3841](https://github.com/apache/datafusion-comet/issues/3841) |
+| `mode` | 🔜 | — |
[#3970](https://github.com/apache/datafusion-comet/issues/3970) |
+| `percentile` | ✅ | Native | Single literal percentage on numeric input runs
natively; array of percentages and a frequency argument fall back to Spark |
+| `percentile_cont` | ✅ | — | Spark 4.0+ `WITHIN GROUP (ORDER BY ...)`;
ascending only runs natively, `DESC` falls back to Spark |
+| `percentile_disc` | 🔜 | — | Percentile aggregate |
+| `regr_avgx` | ✅ | — | Native: Spark rewrites to `Average` (tests in
[#4551](https://github.com/apache/datafusion-comet/issues/4551)) |
+| `regr_avgy` | ✅ | — | Native: Spark rewrites to `Average` (tests in
[#4551](https://github.com/apache/datafusion-comet/issues/4551)) |
+| `regr_count` | ✅ | — | Native: Spark rewrites to `Count` (tests in
[#4551](https://github.com/apache/datafusion-comet/issues/4551)) |
+| `regr_intercept` | 🔜 | — | Falls back; can reuse `covar_pop`/`var_pop`
accumulators ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
+| `regr_r2` | 🔜 | — | Falls back; can reuse the `corr` accumulator
([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
+| `regr_slope` | 🔜 | — | Falls back; can reuse `covar_pop`/`var_pop`
accumulators ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
+| `regr_sxx` | 🔜 | — | Falls back; can reuse `var_pop` accumulator
([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
+| `regr_sxy` | 🔜 | — | Falls back; can reuse `covar_pop` accumulator
([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
+| `regr_syy` | 🔜 | — | Falls back; can reuse `var_pop` accumulator
([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
+| `skewness` | 🔜 | — | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `some` | ✅ | — | |
+| `std` | ✅ | Native | |
+| `stddev` | ✅ | Native | |
+| `stddev_pop` | ✅ | Native | |
+| `stddev_samp` | ✅ | Native | |
+| `string_agg` | 🔜 | — | String aggregation (alias of `listagg`) |
+| `sum` | ✅ | Native | |
+| `try_avg` | ✅ | — | Interval types fall back |
+| `try_sum` | ✅ | — | |
+| `var_pop` | ✅ | Native | |
+| `var_samp` | ✅ | Native | |
+| `variance` | ✅ | Native | |
---
## array_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `array` | ✅ | |
-| `array_append` | ✅ | |
-| `array_compact` | ✅ | |
-| `array_contains` | ✅ | NaN/signed-zero handling may differ
([details](compatibility/floating-point.md)) |
-| `array_distinct` | ✅ | NaN/signed-zero handling may differ
([details](compatibility/floating-point.md)) |
-| `array_except` | ✅ | Routes through the JVM codegen dispatcher by default;
the incompatible native path is opt-in via allowIncompatible
([details](compatibility/expressions/array.md)) |
-| `array_insert` | ✅ | |
-| `array_intersect` | ✅ | Routes through the JVM codegen dispatcher by
default; the incompatible native path is opt-in via allowIncompatible
([details](compatibility/expressions/array.md)) |
-| `array_join` | ✅ | Routes through the JVM codegen dispatcher by default; the
incompatible native path is opt-in via allowIncompatible
([details](compatibility/expressions/array.md)) |
-| `array_max` | ✅ | NaN ordering may differ
([details](compatibility/floating-point.md)) |
-| `array_min` | ✅ | NaN ordering may differ
([details](compatibility/floating-point.md)) |
-| `array_position` | ✅ | Binary/struct/map/null elements fall back |
-| `array_prepend` | ✅ | |
-| `array_remove` | ✅ | |
-| `array_repeat` | ✅ | |
-| `array_union` | ✅ | NaN/signed-zero handling may differ
([details](compatibility/floating-point.md)) |
-| `arrays_overlap` | ✅ | |
-| `arrays_zip` | ✅ | |
-| `element_at` | ✅ | |
-| `flatten` | ✅ | Binary/struct/map elements fall back |
-| `get` | ✅ | |
-| `sequence` | ✅ | |
-| `shuffle` | ✅ | Binary/struct/map elements fall back |
-| `slice` | ✅ | Native
([#4149](https://github.com/apache/datafusion-comet/issues/4149)) |
-| `sort_array` | ✅ | Nested struct/null arrays fall back |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `array` | ✅ | Native | |
+| `array_append` | ✅ | Native | |
+| `array_compact` | ✅ | — | |
+| `array_contains` | ✅ | Native | NaN/signed-zero handling may differ
([details](compatibility/floating-point.md)) |
+| `array_distinct` | ✅ | Native | NaN/signed-zero handling may differ
([details](compatibility/floating-point.md)) |
+| `array_except` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by
default; the incompatible native path is opt-in via allowIncompatible
([details](compatibility/expressions/array.md)) |
+| `array_insert` | ✅ | Native | |
+| `array_intersect` | ✅ | Hybrid | Routes through the JVM codegen dispatcher
by default; the incompatible native path is opt-in via allowIncompatible
([details](compatibility/expressions/array.md)) |
+| `array_join` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by
default; the incompatible native path is opt-in via allowIncompatible
([details](compatibility/expressions/array.md)) |
+| `array_max` | ✅ | Native | NaN ordering may differ
([details](compatibility/floating-point.md)) |
+| `array_min` | ✅ | Native | NaN ordering may differ
([details](compatibility/floating-point.md)) |
+| `array_position` | ✅ | Native | Binary/struct/map/null elements fall back |
+| `array_prepend` | ✅ | — | |
+| `array_remove` | ✅ | Native | |
+| `array_repeat` | ✅ | Native | |
+| `array_union` | ✅ | Native | NaN/signed-zero handling may differ
([details](compatibility/floating-point.md)) |
+| `arrays_overlap` | ✅ | Native | |
+| `arrays_zip` | ✅ | Native | |
+| `element_at` | ✅ | Native | |
+| `flatten` | ✅ | Native | Binary/struct/map elements fall back |
+| `get` | ✅ | — | |
+| `sequence` | ✅ | Codegen dispatch | |
+| `shuffle` | ✅ | Native | Binary/struct/map elements fall back |
+| `slice` | ✅ | Native | Native
([#4149](https://github.com/apache/datafusion-comet/issues/4149)) |
+| `sort_array` | ✅ | Hybrid | Nested struct/null arrays fall back |
---
## bitwise_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `&` | ✅ | |
-| `<<` | ✅ | |
-| `>>` | ✅ | |
-| `>>>` | ✅ | Operator alias for `shiftrightunsigned` (Spark 4.0+) |
-| `^` | ✅ | |
-| `bit_count` | ✅ | |
-| `bit_get` | ✅ | |
-| `getbit` | ✅ | |
-| `shiftright` | ✅ | |
-| `shiftrightunsigned` | ✅ | |
-| `\|` | ✅ | |
-| `~` | ✅ | |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `&` | ✅ | Native | |
+| `<<` | ✅ | — | |
+| `>>` | ✅ | — | |
+| `>>>` | ✅ | — | Operator alias for `shiftrightunsigned` (Spark 4.0+) |
+| `^` | ✅ | Native | |
+| `bit_count` | ✅ | Native | |
+| `bit_get` | ✅ | Native | |
+| `getbit` | ✅ | Native | |
+| `shiftright` | ✅ | Native | |
+| `shiftrightunsigned` | ✅ | Native | |
+| `\|` | ✅ | Native | |
+| `~` | ✅ | Native | |
---
## collection_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `array_size` | ✅ | |
-| `cardinality` | ✅ | |
-| `concat` | ✅ | Binary/array children fall back |
-| `reverse` | ✅ | Binary-element arrays fall back (Incompatible)
([details](compatibility/expressions/array.md)) |
-| `size` | ✅ | |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `array_size` | ✅ | — | |
+| `cardinality` | ✅ | Native | |
+| `concat` | ✅ | Hybrid | Binary/array children fall back |
+| `reverse` | ✅ | Hybrid | Binary-element arrays fall back (Incompatible)
([details](compatibility/expressions/array.md)) |
+| `size` | ✅ | Native | |
---
## conditional_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `coalesce` | ✅ | |
-| `if` | ✅ | |
-| `ifnull` | ✅ | |
-| `nanvl` | ✅ | |
-| `nullif` | ✅ | |
-| `nullifzero` | ✅ | Lowers to `if`/`=` (Spark 4.0+) |
-| `nvl` | ✅ | |
-| `nvl2` | ✅ | |
-| `when` | ✅ | |
-| `zeroifnull` | ✅ | Lowers to `coalesce` (Spark 4.0+) |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `coalesce` | ✅ | Native | |
+| `if` | ✅ | Native | |
+| `ifnull` | ✅ | — | |
+| `nanvl` | ✅ | Codegen dispatch | |
+| `nullif` | ✅ | — | |
+| `nullifzero` | ✅ | — | Lowers to `if`/`=` (Spark 4.0+) |
+| `nvl` | ✅ | — | |
+| `nvl2` | ✅ | — | |
+| `when` | ✅ | Native | |
+| `zeroifnull` | ✅ | — | Lowers to `coalesce` (Spark 4.0+) |
---
@@ -214,99 +227,99 @@ The tables below list every Spark built-in expression
with its current status.
The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`,
`decimal`, `double`, `float`, `int`, `smallint`, `string`, `timestamp`,
`tinyint`) are SQL aliases for `CAST(... AS <type>)` and share the support and
caveats of `cast`.
-| Function | Status | Notes |
-| --- | --- | --- |
-| `cast` | ✅ | Some casts fall back; float-to-decimal is opt-in
([details](compatibility/expressions/cast.md)) |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `cast` | ✅ | Native | Some casts fall back; float-to-decimal is opt-in
([details](compatibility/expressions/cast.md)) |
---
## csv_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `from_csv` | ✅ | |
-| `schema_of_csv` | ✅ | |
-| `to_csv` | ✅ | |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `from_csv` | ✅ | Codegen dispatch | |
+| `schema_of_csv` | ✅ | Codegen dispatch | |
+| `to_csv` | ✅ | Native | |
---
## datetime_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `add_months` | ✅ | |
-| `convert_timezone` | ✅ | Routes through the JVM codegen dispatcher by
default (handles all timezone forms); the native path is opt-in via
allowIncompatible ([details](compatibility/expressions/datetime.md)) |
-| `curdate` | ✅ | Constant-folded to a literal (alias of `current_date`) |
-| `current_date` | ✅ | Constant-folded to a literal before Comet sees the plan
|
-| `current_time` | 🔜 | Blocked on Spark 4.1 TIME type support
([#4288](https://github.com/apache/datafusion-comet/issues/4288)) |
-| `current_timestamp` | ✅ | Constant-folded to a literal before Comet sees the
plan |
-| `current_timezone` | ✅ | |
-| `date_add` | ✅ | |
-| `date_diff` | ✅ | |
-| `date_format` | ✅ | |
-| `date_from_unix_date` | ✅ | |
-| `date_part` | ✅ | |
-| `date_sub` | ✅ | |
-| `date_trunc` | ✅ | |
-| `dateadd` | ✅ | |
-| `datediff` | ✅ | |
-| `datepart` | ✅ | |
-| `day` | ✅ | |
-| `dayname` | ✅ | Abbreviated day name (Spark 4.0+) |
-| `dayofmonth` | ✅ | |
-| `dayofweek` | ✅ | |
-| `dayofyear` | ✅ | |
-| `extract` | ✅ | |
-| `from_unixtime` | ✅ | |
-| `from_utc_timestamp` | ✅ | Routes through the JVM codegen dispatcher by
default (handles all timezone forms); the native path is opt-in via
allowIncompatible ([details](compatibility/expressions/datetime.md)) |
-| `hour` | ✅ | |
-| `last_day` | ✅ | |
-| `localtimestamp` | ✅ | |
-| `make_date` | ✅ | |
-| `make_dt_interval` | ✅ | |
-| `make_interval` | 🔜 | Produces legacy CalendarInterval; tracked by
[#4540](https://github.com/apache/datafusion-comet/issues/4540) |
-| `make_time` | 🔜 | Spark 4.1 TIME type; tracked by
[#4288](https://github.com/apache/datafusion-comet/issues/4288) |
-| `make_timestamp` | ✅ | |
-| `make_timestamp_ltz` | ✅ | 2-arg TIME form falls back |
-| `make_timestamp_ntz` | ✅ | 2-arg TIME form falls back |
-| `make_ym_interval` | ✅ | |
-| `minute` | ✅ | |
-| `month` | ✅ | |
-| `monthname` | ✅ | Abbreviated month name (Spark 4.0+) |
-| `months_between` | ✅ | |
-| `next_day` | ✅ | |
-| `now` | ✅ | Constant-folded to a literal (alias of `current_timestamp`) |
-| `quarter` | ✅ | |
-| `second` | ✅ | |
-| `session_window` | 🔜 | Batch session-window grouping falls back
(`UpdatingSessionsExec` is not yet native); tracked by
[#4785](https://github.com/apache/datafusion-comet/issues/4785) |
-| `time_diff` | 🔜 | Spark 4.1 TIME type; tracked by
[#4288](https://github.com/apache/datafusion-comet/issues/4288) |
-| `time_trunc` | 🔜 | Spark 4.1 TIME type; tracked by
[#4288](https://github.com/apache/datafusion-comet/issues/4288) |
-| `timestamp_micros` | ✅ | |
-| `timestamp_millis` | ✅ | |
-| `timestamp_seconds` | ✅ | |
-| `to_date` | ✅ | Rewrites to `Cast` (or `Cast(GetTimestamp)` with a format)
before Comet sees the plan |
-| `to_time` | 🔜 | Spark 4.1 TIME type; tracked by
[#4288](https://github.com/apache/datafusion-comet/issues/4288) |
-| `to_timestamp` | ✅ | Rewrites to `Cast` (or `GetTimestamp` with a format)
before Comet sees the plan |
-| `to_timestamp_ltz` | ✅ | Rewrites to `to_timestamp` (`TimestampType`) |
-| `to_timestamp_ntz` | ✅ | Rewrites to `to_timestamp` (`TimestampNTZType`) |
-| `to_unix_timestamp` | ✅ | |
-| `to_utc_timestamp` | ✅ | Routes through the JVM codegen dispatcher by
default (handles all timezone forms); the native path is opt-in via
allowIncompatible ([details](compatibility/expressions/datetime.md)) |
-| `trunc` | ✅ | |
-| `try_make_interval` | 🔜 | Produces legacy CalendarInterval; tracked by
[#4540](https://github.com/apache/datafusion-comet/issues/4540) |
-| `try_make_timestamp` | ✅ | |
-| `try_to_date` | ✅ | Rewrites to `Cast`/`GetTimestamp` before Comet sees the
plan; same support as `to_date` |
-| `try_to_time` | 🔜 | Spark 4.1 TIME type; tracked by
[#4288](https://github.com/apache/datafusion-comet/issues/4288) |
-| `try_to_timestamp` | ✅ | Rewrites to `Cast`/`GetTimestamp` before Comet sees
the plan; same support as `to_timestamp` |
-| `unix_date` | ✅ | |
-| `unix_micros` | ✅ | |
-| `unix_millis` | ✅ | |
-| `unix_seconds` | ✅ | |
-| `unix_timestamp` | ✅ | |
-| `weekday` | ✅ | |
-| `weekofyear` | ✅ | |
-| `window` | ✅ | Batch tumbling and sliding time-window grouping runs natively
|
-| `window_time` | ✅ | Batch time-window grouping runs natively |
-| `year` | ✅ | |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `add_months` | ✅ | Codegen dispatch | |
+| `convert_timezone` | ✅ | Hybrid | Routes through the JVM codegen dispatcher
by default (handles all timezone forms); the native path is opt-in via
allowIncompatible ([details](compatibility/expressions/datetime.md)) |
+| `curdate` | ✅ | — | Constant-folded to a literal (alias of `current_date`) |
+| `current_date` | ✅ | — | Constant-folded to a literal before Comet sees the
plan |
+| `current_time` | 🔜 | — | Blocked on Spark 4.1 TIME type support
([#4288](https://github.com/apache/datafusion-comet/issues/4288)) |
+| `current_timestamp` | ✅ | — | Constant-folded to a literal before Comet sees
the plan |
+| `current_timezone` | ✅ | — | |
+| `date_add` | ✅ | Native | |
+| `date_diff` | ✅ | Native | |
+| `date_format` | ✅ | Hybrid | |
+| `date_from_unix_date` | ✅ | Native | |
+| `date_part` | ✅ | — | |
+| `date_sub` | ✅ | Native | |
+| `date_trunc` | ✅ | Hybrid | |
+| `dateadd` | ✅ | Native | |
+| `datediff` | ✅ | Native | |
+| `datepart` | ✅ | — | |
+| `day` | ✅ | Native | |
+| `dayname` | ✅ | — | Abbreviated day name (Spark 4.0+) |
+| `dayofmonth` | ✅ | Native | |
+| `dayofweek` | ✅ | Native | |
+| `dayofyear` | ✅ | Native | |
+| `extract` | ✅ | — | |
+| `from_unixtime` | ✅ | Hybrid | |
+| `from_utc_timestamp` | ✅ | Hybrid | Routes through the JVM codegen
dispatcher by default (handles all timezone forms); the native path is opt-in
via allowIncompatible ([details](compatibility/expressions/datetime.md)) |
+| `hour` | ✅ | Native | |
+| `last_day` | ✅ | Native | |
+| `localtimestamp` | ✅ | — | |
+| `make_date` | ✅ | Native | |
+| `make_dt_interval` | ✅ | Codegen dispatch | |
+| `make_interval` | 🔜 | — | Produces legacy CalendarInterval; tracked by
[#4540](https://github.com/apache/datafusion-comet/issues/4540) |
+| `make_time` | 🔜 | — | Spark 4.1 TIME type; tracked by
[#4288](https://github.com/apache/datafusion-comet/issues/4288) |
+| `make_timestamp` | ✅ | Hybrid | |
+| `make_timestamp_ltz` | ✅ | — | 2-arg TIME form falls back |
+| `make_timestamp_ntz` | ✅ | — | 2-arg TIME form falls back |
+| `make_ym_interval` | ✅ | Codegen dispatch | |
+| `minute` | ✅ | Native | |
+| `month` | ✅ | Native | |
+| `monthname` | ✅ | — | Abbreviated month name (Spark 4.0+) |
+| `months_between` | ✅ | Codegen dispatch | |
+| `next_day` | ✅ | Native | |
+| `now` | ✅ | — | Constant-folded to a literal (alias of `current_timestamp`) |
+| `quarter` | ✅ | Native | |
+| `second` | ✅ | Native | |
+| `session_window` | 🔜 | — | Batch session-window grouping falls back
(`UpdatingSessionsExec` is not yet native); tracked by
[#4785](https://github.com/apache/datafusion-comet/issues/4785) |
+| `time_diff` | 🔜 | — | Spark 4.1 TIME type; tracked by
[#4288](https://github.com/apache/datafusion-comet/issues/4288) |
+| `time_trunc` | 🔜 | — | Spark 4.1 TIME type; tracked by
[#4288](https://github.com/apache/datafusion-comet/issues/4288) |
+| `timestamp_micros` | ✅ | Codegen dispatch | |
+| `timestamp_millis` | ✅ | Codegen dispatch | |
+| `timestamp_seconds` | ✅ | Native | |
+| `to_date` | ✅ | — | Rewrites to `Cast` (or `Cast(GetTimestamp)` with a
format) before Comet sees the plan |
+| `to_time` | 🔜 | — | Spark 4.1 TIME type; tracked by
[#4288](https://github.com/apache/datafusion-comet/issues/4288) |
+| `to_timestamp` | ✅ | — | Rewrites to `Cast` (or `GetTimestamp` with a
format) before Comet sees the plan |
+| `to_timestamp_ltz` | ✅ | — | Rewrites to `to_timestamp` (`TimestampType`) |
+| `to_timestamp_ntz` | ✅ | — | Rewrites to `to_timestamp` (`TimestampNTZType`)
|
+| `to_unix_timestamp` | ✅ | Hybrid | |
+| `to_utc_timestamp` | ✅ | Hybrid | Routes through the JVM codegen dispatcher
by default (handles all timezone forms); the native path is opt-in via
allowIncompatible ([details](compatibility/expressions/datetime.md)) |
+| `trunc` | ✅ | Hybrid | |
+| `try_make_interval` | 🔜 | — | Produces legacy CalendarInterval; tracked by
[#4540](https://github.com/apache/datafusion-comet/issues/4540) |
+| `try_make_timestamp` | ✅ | — | |
+| `try_to_date` | ✅ | — | Rewrites to `Cast`/`GetTimestamp` before Comet sees
the plan; same support as `to_date` |
+| `try_to_time` | 🔜 | — | Spark 4.1 TIME type; tracked by
[#4288](https://github.com/apache/datafusion-comet/issues/4288) |
+| `try_to_timestamp` | ✅ | — | Rewrites to `Cast`/`GetTimestamp` before Comet
sees the plan; same support as `to_timestamp` |
+| `unix_date` | ✅ | Native | |
+| `unix_micros` | ✅ | Codegen dispatch | |
+| `unix_millis` | ✅ | Codegen dispatch | |
+| `unix_seconds` | ✅ | Codegen dispatch | |
+| `unix_timestamp` | ✅ | Native | |
+| `weekday` | ✅ | Native | |
+| `weekofyear` | ✅ | Native | |
+| `window` | ✅ | — | Batch tumbling and sliding time-window grouping runs
natively |
+| `window_time` | ✅ | — | Batch time-window grouping runs natively |
+| `year` | ✅ | Native | |
---
@@ -316,311 +329,311 @@ The type-name conversion functions (`bigint`, `binary`,
`boolean`, `date`, `deci
expression-level). The `outer` variants are wired but marked `Incompatible`;
they require
`spark.comet.exec.explode.enabled=true` and `allowIncompatible`.
-| Function | Status | Notes |
-| --- | --- | --- |
-| `explode` | ✅ | via `CometExplodeExec` |
-| `explode_outer` | ✅ | outer=true falls back (Incompatible)
([audit](../../contributor-guide/expression-audits/generator_funcs.md#explode_outer))
|
-| `inline` | 🔜 | Operator-level generator (like `explode`) |
-| `inline_outer` | 🔜 | Operator-level generator (like `explode`) |
-| `posexplode` | ✅ | via `CometExplodeExec` |
-| `posexplode_outer` | ✅ | outer=true falls back (Incompatible)
([audit](../../contributor-guide/expression-audits/generator_funcs.md#posexplode_outer))
|
-| `stack` | 🔜 | Operator-level generator |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `explode` | ✅ | — | via `CometExplodeExec` |
+| `explode_outer` | ✅ | — | outer=true falls back (Incompatible)
([audit](../../contributor-guide/expression-audits/generator_funcs.md#explode_outer))
|
+| `inline` | 🔜 | — | Operator-level generator (like `explode`) |
+| `inline_outer` | 🔜 | — | Operator-level generator (like `explode`) |
+| `posexplode` | ✅ | — | via `CometExplodeExec` |
+| `posexplode_outer` | ✅ | — | outer=true falls back (Incompatible)
([audit](../../contributor-guide/expression-audits/generator_funcs.md#posexplode_outer))
|
+| `stack` | 🔜 | — | Operator-level generator |
---
## hash_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `crc32` | ✅ | |
-| `hash` | ✅ | |
-| `md5` | ✅ | |
-| `sha` | ✅ | |
-| `sha1` | ✅ | |
-| `sha2` | ✅ | |
-| `xxhash64` | ✅ | |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `crc32` | ✅ | Native | |
+| `hash` | ✅ | Native | |
+| `md5` | ✅ | Native | |
+| `sha` | ✅ | Native | |
+| `sha1` | ✅ | Native | |
+| `sha2` | ✅ | Native | |
+| `xxhash64` | ✅ | Native | |
---
## json_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `from_json` | ✅ | Falls back by default; opt-in via allowIncompatible
([audit](../../contributor-guide/expression-audits/json_funcs.md#from_json)) |
-| `get_json_object` | ✅ | Some inputs need allowIncompatible
([audit](../../contributor-guide/expression-audits/json_funcs.md#get_json_object))
|
-| `json_array_length` | ✅ | Single-quoted/trailing JSON needs
allowIncompatible
([audit](../../contributor-guide/expression-audits/json_funcs.md#json_array_length))
|
-| `json_object_keys` | ✅ | |
-| `json_tuple` | 🔜 |
[#3160](https://github.com/apache/datafusion-comet/issues/3160) |
-| `schema_of_json` | ✅ | |
-| `to_json` | ✅ | Options and map/array inputs fall back
([audit](../../contributor-guide/expression-audits/json_funcs.md#to_json)) |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `from_json` | ✅ | Hybrid | Falls back by default; opt-in via
allowIncompatible
([audit](../../contributor-guide/expression-audits/json_funcs.md#from_json)) |
+| `get_json_object` | ✅ | Hybrid | Some inputs need allowIncompatible
([audit](../../contributor-guide/expression-audits/json_funcs.md#get_json_object))
|
+| `json_array_length` | ✅ | Hybrid | Single-quoted/trailing JSON needs
allowIncompatible
([audit](../../contributor-guide/expression-audits/json_funcs.md#json_array_length))
|
+| `json_object_keys` | ✅ | Codegen dispatch | |
+| `json_tuple` | 🔜 | — |
[#3160](https://github.com/apache/datafusion-comet/issues/3160) |
+| `schema_of_json` | ✅ | Codegen dispatch | |
+| `to_json` | ✅ | Hybrid | Options and map/array inputs fall back
([audit](../../contributor-guide/expression-audits/json_funcs.md#to_json)) |
---
## lambda_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `aggregate` | ✅ | |
-| `array_sort` | ✅ | |
-| `exists` | ✅ | |
-| `filter` | ✅ | General lambda routed through the JVM codegen dispatcher; the
`array_compact` form runs natively |
-| `forall` | ✅ | |
-| `map_filter` | ✅ | |
-| `map_zip_with` | ✅ | |
-| `reduce` | ✅ | |
-| `transform` | ✅ | |
-| `transform_keys` | ✅ | |
-| `transform_values` | ✅ | |
-| `zip_with` | ✅ | |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `aggregate` | ✅ | Codegen dispatch | |
+| `array_sort` | ✅ | Codegen dispatch | |
+| `exists` | ✅ | Codegen dispatch | |
+| `filter` | ✅ | Native | General lambda routed through the JVM codegen
dispatcher; the `array_compact` form runs natively |
+| `forall` | ✅ | Codegen dispatch | |
+| `map_filter` | ✅ | Codegen dispatch | |
+| `map_zip_with` | ✅ | Codegen dispatch | |
+| `reduce` | ✅ | Codegen dispatch | |
+| `transform` | ✅ | Codegen dispatch | |
+| `transform_keys` | ✅ | Codegen dispatch | |
+| `transform_values` | ✅ | Codegen dispatch | |
+| `zip_with` | ✅ | Codegen dispatch | |
---
## map_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `element_at` | ✅ | |
-| `map` | ✅ | Routed through the JVM codegen dispatcher |
-| `map_concat` | ✅ | |
-| `map_contains_key` | ✅ | |
-| `map_entries` | ✅ | |
-| `map_from_arrays` | ✅ | |
-| `map_from_entries` | ✅ | BinaryType key/value falls back (Incompatible)
([details](compatibility/expressions/map.md)) |
-| `map_keys` | ✅ | |
-| `map_values` | ✅ | |
-| `str_to_map` | ✅ | |
-| `try_element_at` | ✅ | Lowers to `element_at` |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `element_at` | ✅ | Native | |
+| `map` | ✅ | Codegen dispatch | Routed through the JVM codegen dispatcher |
+| `map_concat` | ✅ | Codegen dispatch | |
+| `map_contains_key` | ✅ | — | |
+| `map_entries` | ✅ | Native | |
+| `map_from_arrays` | ✅ | Native | |
+| `map_from_entries` | ✅ | Hybrid | BinaryType key/value falls back
(Incompatible) ([details](compatibility/expressions/map.md)) |
+| `map_keys` | ✅ | Native | |
+| `map_values` | ✅ | Native | |
+| `str_to_map` | ✅ | Hybrid | |
+| `try_element_at` | ✅ | — | Lowers to `element_at` |
---
## math_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `%` | ✅ | |
-| `*` | ✅ | Interval multiplication falls back |
-| `+` | ✅ | |
-| `-` | ✅ | |
-| `/` | ✅ | |
-| `abs` | ✅ | Interval types fall back |
-| `acos` | ✅ | |
-| `acosh` | ✅ | |
-| `asin` | ✅ | |
-| `asinh` | ✅ | |
-| `atan` | ✅ | |
-| `atan2` | ✅ | |
-| `atanh` | ✅ | |
-| `bin` | ✅ | |
-| `bround` | ✅ | |
-| `cbrt` | ✅ | |
-| `ceil` | ✅ | Two-arg form falls back |
-| `ceiling` | ✅ | |
-| `conv` | ✅ | |
-| `cos` | ✅ | |
-| `cosh` | ✅ | |
-| `cot` | ✅ | |
-| `csc` | ✅ | |
-| `degrees` | ✅ | |
-| `div` | ✅ | |
-| `e` | ✅ | Folds to a literal (like `pi`) |
-| `exp` | ✅ | |
-| `expm1` | ✅ | |
-| `factorial` | ✅ | |
-| `floor` | ✅ | Two-arg form falls back |
-| `greatest` | ✅ | |
-| `hex` | ✅ | |
-| `hypot` | ✅ | |
-| `least` | ✅ | |
-| `ln` | ✅ | |
-| `log` | ✅ | |
-| `log10` | ✅ | |
-| `log1p` | ✅ | |
-| `log2` | ✅ | |
-| `mod` | ✅ | |
-| `negative` | ✅ | |
-| `pi` | ✅ | |
-| `pmod` | ✅ | |
-| `positive` | ✅ | |
-| `pow` | ✅ | |
-| `power` | ✅ | |
-| `radians` | ✅ | |
-| `rand` | ✅ | |
-| `randn` | ✅ | |
-| `random` | ✅ | Alias for `rand` (Spark 4.0+); seed must be a literal |
-| `randstr` | 🔜 | Random string (Spark 4.0+) |
-| `rint` | ✅ | |
-| `round` | ✅ | Float/double inputs fall back |
-| `sec` | ✅ | |
-| `shiftleft` | ✅ | |
-| `sign` | ✅ | |
-| `signum` | ✅ | |
-| `sin` | ✅ | |
-| `sinh` | ✅ | |
-| `sqrt` | ✅ | |
-| `tan` | ✅ | |
-| `tanh` | ✅ | |
-| `try_add` | ✅ | Datetime/interval form falls back |
-| `try_divide` | ✅ | |
-| `try_mod` | ✅ | |
-| `try_multiply` | ✅ | |
-| `try_subtract` | ✅ | |
-| `unhex` | ✅ | |
-| `uniform` | ✅ | Constant-folded; literal arguments only (Spark 4.0+) |
-| `width_bucket` | ✅ | |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `%` | ✅ | Native | |
+| `*` | ✅ | Native | Interval multiplication falls back |
+| `+` | ✅ | Native | |
+| `-` | ✅ | Native | |
+| `/` | ✅ | Native | |
+| `abs` | ✅ | Native | Interval types fall back |
+| `acos` | ✅ | Native | |
+| `acosh` | ✅ | Native | |
+| `asin` | ✅ | Native | |
+| `asinh` | ✅ | Native | |
+| `atan` | ✅ | Native | |
+| `atan2` | ✅ | Native | |
+| `atanh` | ✅ | Native | |
+| `bin` | ✅ | Native | |
+| `bround` | ✅ | Codegen dispatch | |
+| `cbrt` | ✅ | Native | |
+| `ceil` | ✅ | — | Two-arg form falls back |
+| `ceiling` | ✅ | — | |
+| `conv` | ✅ | Codegen dispatch | |
+| `cos` | ✅ | Native | |
+| `cosh` | ✅ | Native | |
+| `cot` | ✅ | Native | |
+| `csc` | ✅ | Native | |
+| `degrees` | ✅ | Native | |
+| `div` | ✅ | Native | |
+| `e` | ✅ | — | Folds to a literal (like `pi`) |
+| `exp` | ✅ | Native | |
+| `expm1` | ✅ | Native | |
+| `factorial` | ✅ | Native | |
+| `floor` | ✅ | — | Two-arg form falls back |
+| `greatest` | ✅ | Native | |
+| `hex` | ✅ | Native | |
+| `hypot` | ✅ | Codegen dispatch | |
+| `least` | ✅ | Native | |
+| `ln` | ✅ | Native | |
+| `log` | ✅ | Native | |
+| `log10` | ✅ | Native | |
+| `log1p` | ✅ | Codegen dispatch | |
+| `log2` | ✅ | Native | |
+| `mod` | ✅ | Native | |
+| `negative` | ✅ | Native | |
+| `pi` | ✅ | Native | |
+| `pmod` | ✅ | Codegen dispatch | |
+| `positive` | ✅ | Codegen dispatch | |
+| `pow` | ✅ | Native | |
+| `power` | ✅ | Native | |
+| `radians` | ✅ | Native | |
+| `rand` | ✅ | Native | |
+| `randn` | ✅ | Native | |
+| `random` | ✅ | Native | Alias for `rand` (Spark 4.0+); seed must be a
literal |
+| `randstr` | 🔜 | — | Random string (Spark 4.0+) |
+| `rint` | ✅ | Native | |
+| `round` | ✅ | Native | Float/double inputs fall back |
+| `sec` | ✅ | Native | |
+| `shiftleft` | ✅ | Native | |
+| `sign` | ✅ | Native | |
+| `signum` | ✅ | Native | |
+| `sin` | ✅ | Native | |
+| `sinh` | ✅ | Native | |
+| `sqrt` | ✅ | Native | |
+| `tan` | ✅ | Native | |
+| `tanh` | ✅ | Native | |
+| `try_add` | ✅ | — | Datetime/interval form falls back |
+| `try_divide` | ✅ | — | |
+| `try_mod` | ✅ | — | |
+| `try_multiply` | ✅ | — | |
+| `try_subtract` | ✅ | — | |
+| `unhex` | ✅ | Native | |
+| `uniform` | ✅ | — | Constant-folded; literal arguments only (Spark 4.0+) |
+| `width_bucket` | ✅ | Codegen dispatch | |
---
## misc_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `aes_decrypt` | ✅ | Routed through the JVM codegen dispatcher |
-| `aes_encrypt` | ✅ | Routed through the JVM codegen dispatcher;
nondeterministic IV by default |
-| `assert_true` | 🔜 | Lowers to `RaiseError`, which falls back |
-| `current_catalog` | ✅ | Resolved to a literal by the analyzer
(`ReplaceCurrentLike`) |
-| `current_database` | ✅ | Resolved to a literal by the analyzer
(`ReplaceCurrentLike`) |
-| `current_schema` | ✅ | Alias of `current_database`; resolved to a literal by
the analyzer |
-| `current_user` | ✅ | Resolved to a literal by the analyzer; same as `user` |
-| `equal_null` | ✅ | Lowers to `<=>` (`EqualNullSafe`) |
-| `is_variant_null` | 🔜 | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
-| `monotonically_increasing_id` | ✅ | |
-| `parse_json` | 🔜 | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
-| `raise_error` | 🔜 | Raises a runtime error |
-| `rand` | ✅ | Seed must be a literal |
-| `randn` | ✅ | Seed must be a literal |
-| `schema_of_variant` | 🔜 | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
-| `schema_of_variant_agg` | 🔜 | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
-| `session_user` | ✅ | Alias of `current_user`; resolved to a literal by the
analyzer |
-| `spark_partition_id` | ✅ | |
-| `to_variant_object` | 🔜 | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
-| `try_aes_decrypt` | ✅ | Routed through the JVM codegen dispatcher |
-| `try_parse_json` | 🔜 | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
-| `try_variant_get` | 🔜 | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
-| `typeof` | ✅ | Foldable; resolved to a literal before Comet sees the plan |
-| `user` | ✅ | Resolved to a literal by the Spark analyzer before reaching
Comet |
-| `uuid` | 🔜 | Nondeterministic random UUID |
-| `variant_get` | 🔜 | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `aes_decrypt` | ✅ | — | Routed through the JVM codegen dispatcher |
+| `aes_encrypt` | ✅ | — | Routed through the JVM codegen dispatcher;
nondeterministic IV by default |
+| `assert_true` | 🔜 | — | Lowers to `RaiseError`, which falls back |
+| `current_catalog` | ✅ | — | Resolved to a literal by the analyzer
(`ReplaceCurrentLike`) |
+| `current_database` | ✅ | — | Resolved to a literal by the analyzer
(`ReplaceCurrentLike`) |
+| `current_schema` | ✅ | — | Alias of `current_database`; resolved to a
literal by the analyzer |
+| `current_user` | ✅ | — | Resolved to a literal by the analyzer; same as
`user` |
+| `equal_null` | ✅ | — | Lowers to `<=>` (`EqualNullSafe`) |
+| `is_variant_null` | 🔜 | — | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `monotonically_increasing_id` | ✅ | Native | |
+| `parse_json` | 🔜 | — | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `raise_error` | 🔜 | — | Raises a runtime error |
+| `rand` | ✅ | Native | Seed must be a literal |
+| `randn` | ✅ | Native | Seed must be a literal |
+| `schema_of_variant` | 🔜 | — | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `schema_of_variant_agg` | 🔜 | — | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `session_user` | ✅ | — | Alias of `current_user`; resolved to a literal by
the analyzer |
+| `spark_partition_id` | ✅ | Native | |
+| `to_variant_object` | 🔜 | — | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `try_aes_decrypt` | ✅ | — | Routed through the JVM codegen dispatcher |
+| `try_parse_json` | 🔜 | — | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `try_variant_get` | 🔜 | — | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `typeof` | ✅ | — | Foldable; resolved to a literal before Comet sees the
plan |
+| `user` | ✅ | — | Resolved to a literal by the Spark analyzer before reaching
Comet |
+| `uuid` | 🔜 | — | Nondeterministic random UUID |
+| `variant_get` | 🔜 | — | tracking
[#4098](https://github.com/apache/datafusion-comet/issues/4098) |
---
## predicate_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `!` | ✅ | |
-| `<` | ✅ | |
-| `<=` | ✅ | |
-| `<=>` | ✅ | |
-| `=` | ✅ | |
-| `==` | ✅ | |
-| `>` | ✅ | |
-| `>=` | ✅ | |
-| `and` | ✅ | |
-| `between` | ✅ | |
-| `ilike` | ✅ | |
-| `in` | ✅ | |
-| `isnan` | ✅ | |
-| `isnotnull` | ✅ | |
-| `isnull` | ✅ | |
-| `like` | ✅ | |
-| `not` | ✅ | |
-| `or` | ✅ | |
-| `regexp` | ✅ | Falls back by default; opt-in via allowIncompatible
([details](compatibility/regex.md)) |
-| `regexp_like` | ✅ | Falls back by default; opt-in via allowIncompatible
([details](compatibility/regex.md)) |
-| `rlike` | ✅ | Falls back by default; opt-in via allowIncompatible
([details](compatibility/regex.md)) |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `!` | ✅ | Native | |
+| `<` | ✅ | Hybrid | |
+| `<=` | ✅ | Hybrid | |
+| `<=>` | ✅ | Hybrid | |
+| `=` | ✅ | Hybrid | |
+| `==` | ✅ | Hybrid | |
+| `>` | ✅ | Hybrid | |
+| `>=` | ✅ | Hybrid | |
+| `and` | ✅ | Native | |
+| `between` | ✅ | — | |
+| `ilike` | ✅ | — | |
+| `in` | ✅ | Hybrid | |
+| `isnan` | ✅ | Native | |
+| `isnotnull` | ✅ | Native | |
+| `isnull` | ✅ | Native | |
+| `like` | ✅ | Hybrid | |
+| `not` | ✅ | Native | |
+| `or` | ✅ | Native | |
+| `regexp` | ✅ | Hybrid | Falls back by default; opt-in via allowIncompatible
([details](compatibility/regex.md)) |
+| `regexp_like` | ✅ | Hybrid | Falls back by default; opt-in via
allowIncompatible ([details](compatibility/regex.md)) |
+| `rlike` | ✅ | Hybrid | Falls back by default; opt-in via allowIncompatible
([details](compatibility/regex.md)) |
---
## string_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `ascii` | ✅ | |
-| `base64` | ✅ | |
-| `bit_length` | ✅ | |
-| `btrim` | ✅ | |
-| `char` | ✅ | |
-| `char_length` | ✅ | |
-| `character_length` | ✅ | |
-| `chr` | ✅ | |
-| `collate` | 🔜 | Spark collation (umbrella
[#2190](https://github.com/apache/datafusion-comet/issues/2190)) |
-| `collation` | ✅ | Constant-folded to a literal (Spark 4.0+) |
-| `concat_ws` | ✅ | |
-| `contains` | ✅ | |
-| `decode` | ✅ | |
-| `elt` | ✅ | |
-| `encode` | 🔜 | Lowers to `StaticInvoke(encode)` (not allowlisted); falls
back |
-| `endswith` | ✅ | |
-| `find_in_set` | ✅ | |
-| `format_number` | ✅ | |
-| `format_string` | ✅ | |
-| `initcap` | ✅ | |
-| `instr` | ✅ | |
-| `lcase` | ✅ | |
-| `left` | ✅ | |
-| `len` | ✅ | |
-| `length` | ✅ | |
-| `levenshtein` | ✅ | |
-| `locate` | ✅ | |
-| `lower` | ✅ | |
-| `lpad` | ✅ | |
-| `ltrim` | ✅ | |
-| `luhn_check` | ✅ | Native via `StaticInvoke` (tests: luhn_check.sql) |
-| `mask` | ✅ | Routed through the JVM codegen dispatcher |
-| `octet_length` | ✅ | |
-| `overlay` | ✅ | |
-| `position` | ✅ | |
-| `printf` | ✅ | |
-| `regexp_count` | ✅ | Runs natively (rewrites to
`size(regexp_extract_all(...))`) |
-| `regexp_extract` | ✅ | |
-| `regexp_extract_all` | ✅ | |
-| `regexp_instr` | ✅ | Routed through the JVM codegen dispatcher |
-| `regexp_replace` | ✅ | |
-| `regexp_substr` | ✅ | Runs natively (rewrites to
`nullif(regexp_extract(...), '')`) |
-| `repeat` | ✅ | |
-| `replace` | ✅ | |
-| `right` | ✅ | |
-| `rpad` | ✅ | |
-| `rtrim` | ✅ | |
-| `soundex` | ✅ | |
-| `space` | ✅ | |
-| `split` | ✅ | |
-| `split_part` | ✅ | Spark 4.0+ |
-| `startswith` | ✅ | |
-| `substr` | ✅ | |
-| `substring` | ✅ | |
-| `substring_index` | ✅ | |
-| `to_binary` | ✅ | Hex form accelerated; other formats fall back |
-| `to_char` | ✅ | |
-| `to_number` | ✅ | |
-| `to_varchar` | ✅ | |
-| `translate` | ✅ | Falls back by default; opt-in via allowIncompatible
([#4463](https://github.com/apache/datafusion-comet/issues/4463)) |
-| `trim` | ✅ | |
-| `try_to_binary` | ✅ | Runs natively (rewrites to `try_eval(to_binary(...))`)
|
-| `try_to_number` | ✅ | Routed through the JVM codegen dispatcher |
-| `ucase` | ✅ | |
-| `unbase64` | ✅ | |
-| `upper` | ✅ | |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `ascii` | ✅ | Native | |
+| `base64` | ✅ | Native | |
+| `bit_length` | ✅ | Native | |
+| `btrim` | ✅ | — | |
+| `char` | ✅ | Native | |
+| `char_length` | ✅ | Native | |
+| `character_length` | ✅ | Native | |
+| `chr` | ✅ | Native | |
+| `collate` | 🔜 | — | Spark collation (umbrella
[#2190](https://github.com/apache/datafusion-comet/issues/2190)) |
+| `collation` | ✅ | — | Constant-folded to a literal (Spark 4.0+) |
+| `concat_ws` | ✅ | Native | |
+| `contains` | ✅ | — | |
+| `decode` | ✅ | — | |
+| `elt` | ✅ | Codegen dispatch | |
+| `encode` | 🔜 | — | Lowers to `StaticInvoke(encode)` (not allowlisted); falls
back |
+| `endswith` | ✅ | — | |
+| `find_in_set` | ✅ | Codegen dispatch | |
+| `format_number` | ✅ | Codegen dispatch | |
+| `format_string` | ✅ | Codegen dispatch | |
+| `initcap` | ✅ | Hybrid | |
+| `instr` | ✅ | Native | |
+| `lcase` | ✅ | Hybrid | |
+| `left` | ✅ | Native | |
+| `len` | ✅ | Native | |
+| `length` | ✅ | Native | |
+| `levenshtein` | ✅ | Native | |
+| `locate` | ✅ | Codegen dispatch | |
+| `lower` | ✅ | Hybrid | |
+| `lpad` | ✅ | — | |
+| `ltrim` | ✅ | Native | |
+| `luhn_check` | ✅ | — | Native via `StaticInvoke` (tests: luhn_check.sql) |
+| `mask` | ✅ | — | Routed through the JVM codegen dispatcher |
+| `octet_length` | ✅ | Native | |
+| `overlay` | ✅ | Codegen dispatch | |
+| `position` | ✅ | Codegen dispatch | |
+| `printf` | ✅ | Codegen dispatch | |
+| `regexp_count` | ✅ | — | Runs natively (rewrites to
`size(regexp_extract_all(...))`) |
+| `regexp_extract` | ✅ | Native | |
+| `regexp_extract_all` | ✅ | Native | |
+| `regexp_instr` | ✅ | Codegen dispatch | Routed through the JVM codegen
dispatcher |
+| `regexp_replace` | ✅ | Hybrid | |
+| `regexp_substr` | ✅ | — | Runs natively (rewrites to
`nullif(regexp_extract(...), '')`) |
+| `repeat` | ✅ | Native | |
+| `replace` | ✅ | Hybrid | |
+| `right` | ✅ | Native | |
+| `rpad` | ✅ | — | |
+| `rtrim` | ✅ | Native | |
+| `soundex` | ✅ | Native | |
+| `space` | ✅ | Native | |
+| `split` | ✅ | Hybrid | |
+| `split_part` | ✅ | — | Spark 4.0+ |
+| `startswith` | ✅ | — | |
+| `substr` | ✅ | Native | |
+| `substring` | ✅ | Native | |
+| `substring_index` | ✅ | Native | |
+| `to_binary` | ✅ | — | Hex form accelerated; other formats fall back |
+| `to_char` | ✅ | Codegen dispatch | |
+| `to_number` | ✅ | Codegen dispatch | |
+| `to_varchar` | ✅ | Codegen dispatch | |
+| `translate` | ✅ | Native | Falls back by default; opt-in via
allowIncompatible
([#4463](https://github.com/apache/datafusion-comet/issues/4463)) |
+| `trim` | ✅ | Native | |
+| `try_to_binary` | ✅ | — | Runs natively (rewrites to
`try_eval(to_binary(...))`) |
+| `try_to_number` | ✅ | Codegen dispatch | Routed through the JVM codegen
dispatcher |
+| `ucase` | ✅ | Hybrid | |
+| `unbase64` | ✅ | Codegen dispatch | |
+| `upper` | ✅ | Hybrid | |
---
## struct_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `named_struct` | ✅ | Duplicate field names fall back |
-| `struct` | ✅ | |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `named_struct` | ✅ | Native | Duplicate field names fall back |
+| `struct` | ✅ | Native | |
---
## url_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `parse_url` | ✅ | |
-| `try_url_decode` | ✅ | |
-| `url_decode` | ✅ | |
-| `url_encode` | ✅ | |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `parse_url` | ✅ | Native | |
+| `try_url_decode` | ✅ | — | |
+| `url_decode` | ✅ | — | |
+| `url_encode` | ✅ | — | |
---
@@ -637,36 +650,36 @@ to Spark when used as window functions. A handful of
frame shapes also
fall back. See [window function compatibility](compatibility/operators.md)
for the full list of supported functions, frames, and fallback cases.
-| Function | Status | Notes |
-| --- | --- | --- |
-| `cume_dist` | ✅ | via `CometWindowExec` |
-| `dense_rank` | ✅ | via `CometWindowExec` |
-| `lag` | ✅ | via `CometWindowExec`; non-literal default falls back
([#4268](https://github.com/apache/datafusion-comet/issues/4268)) |
-| `lead` | ✅ | via `CometWindowExec`; non-literal default falls back
([#4268](https://github.com/apache/datafusion-comet/issues/4268)) |
-| `nth_value` | ✅ | via `CometWindowExec` |
-| `ntile` | ✅ | via `CometWindowExec` |
-| `percent_rank` | ✅ | via `CometWindowExec` |
-| `rank` | ✅ | via `CometWindowExec` |
-| `row_number` | ✅ | via `CometWindowExec` |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `cume_dist` | ✅ | — | via `CometWindowExec` |
+| `dense_rank` | ✅ | — | via `CometWindowExec` |
+| `lag` | ✅ | — | via `CometWindowExec`; non-literal default falls back
([#4268](https://github.com/apache/datafusion-comet/issues/4268)) |
+| `lead` | ✅ | — | via `CometWindowExec`; non-literal default falls back
([#4268](https://github.com/apache/datafusion-comet/issues/4268)) |
+| `nth_value` | ✅ | — | via `CometWindowExec` |
+| `ntile` | ✅ | — | via `CometWindowExec` |
+| `percent_rank` | ✅ | — | via `CometWindowExec` |
+| `rank` | ✅ | — | via `CometWindowExec` |
+| `row_number` | ✅ | — | via `CometWindowExec` |
---
## xml_funcs
-| Function | Status | Notes |
-| --- | --- | --- |
-| `from_xml` | ✅ | Spark 4.0+ |
-| `schema_of_xml` | ✅ | Spark 4.0+ |
-| `to_xml` | ✅ | Spark 4.0+ |
-| `xpath` | ✅ | |
-| `xpath_boolean` | ✅ | |
-| `xpath_double` | ✅ | |
-| `xpath_float` | ✅ | |
-| `xpath_int` | ✅ | |
-| `xpath_long` | ✅ | |
-| `xpath_number` | ✅ | Alias of `xpath_double` |
-| `xpath_short` | ✅ | |
-| `xpath_string` | ✅ | |
+| Function | Status | Implementation | Notes |
+| --- | --- | --- | --- |
+| `from_xml` | ✅ | — | Spark 4.0+ |
+| `schema_of_xml` | ✅ | — | Spark 4.0+ |
+| `to_xml` | ✅ | — | Spark 4.0+ |
+| `xpath` | ✅ | Codegen dispatch | |
+| `xpath_boolean` | ✅ | Codegen dispatch | |
+| `xpath_double` | ✅ | Codegen dispatch | |
+| `xpath_float` | ✅ | Codegen dispatch | |
+| `xpath_int` | ✅ | Codegen dispatch | |
+| `xpath_long` | ✅ | Codegen dispatch | |
+| `xpath_number` | ✅ | Codegen dispatch | Alias of `xpath_double` |
+| `xpath_short` | ✅ | Codegen dispatch | |
+| `xpath_string` | ✅ | Codegen dispatch | |
---
diff --git a/spark/src/main/scala/org/apache/comet/GenerateDocs.scala
b/spark/src/main/scala/org/apache/comet/GenerateDocs.scala
index 9ea512c40c..816dbc6790 100644
--- a/spark/src/main/scala/org/apache/comet/GenerateDocs.scala
+++ b/spark/src/main/scala/org/apache/comet/GenerateDocs.scala
@@ -24,11 +24,12 @@ import java.io.{BufferedOutputStream, BufferedReader,
FileOutputStream, FileRead
import scala.collection.mutable
import scala.collection.mutable.ListBuffer
+import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
import org.apache.spark.sql.catalyst.expressions.Cast
import org.apache.comet.CometConf.COMET_ONHEAP_MEMORY_OVERHEAD
import org.apache.comet.expressions.{CometCast, CometEvalMode}
-import org.apache.comet.serde.{CodegenDispatchFallback,
CometAggregateExpressionSerde, CometExpressionSerde, Compatible, Incompatible,
NativeOptInAvailable, QueryPlanSerde, Unsupported}
+import org.apache.comet.serde.{CodegenDispatchFallback,
CometAggregateExpressionSerde, CometCodegenDispatch, CometExpressionSerde,
Compatible, Incompatible, NativeOptInAvailable, QueryPlanSerde, Unsupported}
/**
* Utility for generating markdown documentation from the configs.
@@ -169,11 +170,132 @@ object GenerateDocs {
}
generateConfigReference(s"$userGuideLocation/configs.md")
generateCompatibilityGuide(s"$compatPagesDir/cast.md")
+ updateExpressionsPageImplementation(s"$userGuideLocation/expressions.md")
for ((category, (filename, notesFn)) <- categoryPages) {
generateExpressionCompatNotes(s"$compatPagesDir/$filename", category,
notesFn())
}
}
+ private val ImplNative = "Native"
+ private val ImplCodegen = "Codegen dispatch"
+ private val ImplHybrid = "Hybrid"
+ private val ImplUnknown: String = new String(Array(8212.toChar))
+
+ /**
+ * Classify an expression serde by which execution path it exposes.
+ *
+ * - `NativeOptInAvailable` (and its subtype `CodegenDispatchFallback`)
means the serde has
+ * both a native path and a Spark-compatible (typically
codegen-dispatch) path.
+ * - A plain `CometCodegenDispatch` means the serde has only a
codegen-dispatch path.
+ * - Everything else is a plain native serde.
+ */
+ private def classifySerde(serde: CometExpressionSerde[_]): String = {
+ if (serde.isInstanceOf[NativeOptInAvailable]) ImplHybrid
+ else if (serde.isInstanceOf[CometCodegenDispatch[_]]) ImplCodegen
+ else ImplNative
+ }
+
+ /**
+ * Build a map from Spark built-in SQL function name (e.g. `array_max`) to
its implementation
+ * kind, resolving each name through Spark's `FunctionRegistry` to the
backing Catalyst
+ * expression class and then looking that class up in `QueryPlanSerde`'s
serde maps.
+ */
+ private def buildFunctionNameToKind(): Map[String, String] = {
+ val classNameToKind: Map[String, String] = {
+ val exprKinds = QueryPlanSerde.exprSerdeMap.iterator.map { case (cls,
serde) =>
+ cls.getName -> classifySerde(serde)
+ }
+ // Aggregate serdes have no codegen-dispatch or opt-in path; they are
always native.
+ val aggKinds = QueryPlanSerde.aggrSerdeMap.iterator.map { case (cls, _)
=>
+ cls.getName -> ImplNative
+ }
+ (exprKinds ++ aggKinds).toMap
+ }
+ FunctionRegistry.expressions.iterator.map { case (name, (info, _)) =>
+ name -> classNameToKind.getOrElse(info.getClassName, ImplUnknown)
+ }.toMap
+ }
+
+ /**
+ * Update the Implementation column of every markdown table in
`expressions.md` in place.
+ *
+ * The generator recognises table blocks by their header row (`| Function |
Status |
+ * Implementation | Notes |`), skips the separator row, and rewrites the
Implementation cell of
+ * each data row based on the function name in the first column. Rows whose
function name is
+ * either not registered in Spark's `FunctionRegistry` or has no serde in
`QueryPlanSerde`
+ * (planned, rewritten to another expression, or operator-level) get the
em-dash placeholder.
+ */
+ private def updateExpressionsPageImplementation(filename: String): Unit = {
+ val nameToKind = buildFunctionNameToKind()
+ val lines = readFileVerbatim(filename)
+ val w = new BufferedOutputStream(new FileOutputStream(filename))
+ var implColumnIdx = -1
+ var afterSeparator = false
+ for (line <- lines) {
+ val trimmed = line.trim
+ if (trimmed.startsWith("|")) {
+ val cells = splitTableRow(trimmed)
+ val headerIdx = cells.indexWhere(_.trim == "Implementation")
+ val isHeader = headerIdx >= 0 && cells.exists(_.trim == "Function") &&
+ cells.exists(_.trim == "Status") && cells.exists(_.trim == "Notes")
+ val isSeparator =
+ implColumnIdx >= 0 && cells.length > 1 &&
+ cells.drop(1).dropRight(1).forall(c => c.trim.matches("-+"))
+ if (isHeader) {
+ implColumnIdx = headerIdx
+ afterSeparator = false
+ w.write(s"${line.stripTrailing()}\n".getBytes)
+ } else if (isSeparator) {
+ afterSeparator = true
+ w.write(s"${line.stripTrailing()}\n".getBytes)
+ } else if (afterSeparator && implColumnIdx >= 0 && cells.length >
implColumnIdx) {
+ // Data row: extract SQL function name from column 1 (0-indexed
cells: [empty, name,
+ // status, impl, notes, empty]). Unescape any `\|` back to `|` since
markdown requires
+ // pipes inside table cells to be backslash-escaped.
+ val nameCell = cells(1).trim
+ val name = nameCell.stripPrefix("`").stripSuffix("`").replace("\\|",
"|")
+ val kind = nameToKind.getOrElse(name, ImplUnknown)
+ val updated = cells.updated(implColumnIdx, s" $kind ")
+ w.write(s"${updated.mkString("|").stripTrailing()}\n".getBytes)
+ } else {
+ w.write(s"${line.stripTrailing()}\n".getBytes)
+ }
+ } else {
+ // Any non-table line ends the current table block.
+ implColumnIdx = -1
+ afterSeparator = false
+ w.write(s"${line.stripTrailing()}\n".getBytes)
+ }
+ }
+ w.close()
+ }
+
+ /**
+ * Split a markdown table row on `|`. Ordinary `String.split` drops trailing
empty tokens, so `|
+ * a | b |` would come back as three elements instead of four. Passing `-1`
as the limit
+ * preserves them, which we need so that column indices stay aligned with
the header row.
+ *
+ * `\|` (the markdown escape for a literal pipe inside a cell) is
temporarily replaced with a
+ * placeholder before splitting so the escaped pipe is not treated as a
column boundary.
+ */
+ private def splitTableRow(row: String): Array[String] = {
+ val placeholder = "\u0001"
+ row.replace("\\|", placeholder).split("\\|",
-1).map(_.replace(placeholder, "\\|"))
+ }
+
+ /** Read a file without stripping the BEGIN/END autogen markers used
elsewhere. */
+ private def readFileVerbatim(filename: String): Seq[String] = {
+ val r = new BufferedReader(new FileReader(filename))
+ val buffer = new ListBuffer[String]()
+ var line = r.readLine()
+ while (line != null) {
+ buffer += line
+ line = r.readLine()
+ }
+ r.close()
+ buffer.toSeq
+ }
+
private def generateConfigReference(filename: String): Unit = {
val pattern = "<!--BEGIN:CONFIG_TABLE\\[(.*)]-->".r
val lines = readFile(filename)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
