This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new d3f9232ee Publish built docs triggered by
8a8ab244101506e4b006944c465bc593604239f4
d3f9232ee is described below
commit d3f9232ee94e2facf638ac246426db8863542898
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Tue Sep 9 16:36:36 2025 +0000
Publish built docs triggered by 8a8ab244101506e4b006944c465bc593604239f4
---
_sources/user-guide/latest/datatypes.md.txt | 44 +-
_sources/user-guide/latest/expressions.md.txt | 409 +++++++-------
_sources/user-guide/latest/operators.md.txt | 34 +-
searchindex.js | 2 +-
user-guide/latest/datatypes.html | 76 +--
user-guide/latest/expressions.html | 781 +++++++++++++++++---------
user-guide/latest/operators.html | 60 +-
7 files changed, 855 insertions(+), 551 deletions(-)
diff --git a/_sources/user-guide/latest/datatypes.md.txt
b/_sources/user-guide/latest/datatypes.md.txt
index 3b7940dd6..8cb5a25ae 100644
--- a/_sources/user-guide/latest/datatypes.md.txt
+++ b/_sources/user-guide/latest/datatypes.md.txt
@@ -19,27 +19,29 @@
# Supported Spark Data Types
-The following Spark data types are currently available:
+Comet supports the following Spark data types. Refer to the [Comet
Compatibility Guide] for information about data
+type support in scans and other operators.
+
+[Comet Compatibility Guide]: compatibility.md
<!-- based on org.apache.comet.serde.QueryPlanSerde.supportedDataType -->
-- Primitives
- - Boolean
- - Byte
- - Short
- - Integer
- - Long
- - Float
- - Double
-- String
-- Binary
-- Decimal
-- Temporal
- - Date
- - Timestamp
- - TimestampNTZ
-- Null
-- Complex Types
- - Struct
- - Array
- - Map
+| Data Type |
+| ------------ |
+| Null |
+| Boolean |
+| Byte |
+| Short |
+| Integer |
+| Long |
+| Float |
+| Double |
+| Decimal |
+| String |
+| Binary |
+| Date |
+| Timestamp |
+| TimestampNTZ |
+| Struct |
+| Array |
+| Map |
diff --git a/_sources/user-guide/latest/expressions.md.txt
b/_sources/user-guide/latest/expressions.md.txt
index 5c7a48403..2746b02ff 100644
--- a/_sources/user-guide/latest/expressions.md.txt
+++ b/_sources/user-guide/latest/expressions.md.txt
@@ -19,227 +19,238 @@
# Supported Spark Expressions
-The following Spark expressions are currently available. Any known
compatibility issues are noted in the following tables.
+Comet supports the following Spark expressions. Expressions that are marked as
Spark-compatible will either run
+natively in Comet and provide the same results as Spark, or will fall back to
Spark for cases that would not
+be compatible.
-## Literal Values
-
-| Expression | Notes |
-| -------------------------------------- | ----- |
-| Literal values of supported data types | |
-
-## Unary Arithmetic
-
-| Expression | Notes |
-| ---------------- | ----- |
-| UnaryMinus (`-`) | |
-
-## Binary Arithmetic
-
-| Expression | Notes
|
-|------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Add (`+`) |
|
-| Subtract (`-`) |
|
-| Multiply (`*`) |
|
-| Divide (`/`) |
|
-| IntegralDivide (`div`) | All operands are cast to DecimalType (in case the
input type is not already decima type) with precision 19 and scale 0. Please
set `spark.comet.expression.allowIncompatible` to `true` to enable DataFusion’s
cast operation for LongType inputs. |
-| Remainder (`%`) |
|
-
-## Binary Try Arithmetic
-
-| Expression | Notes
|
-|------------|----------------------------------------------------------------------------|
-| `try_add` | Adds operands (IntegerTypes only) or results NULL incase of
overflow |
-| `try_sub` | Subtracts operands (IntegerTypes only) or results NULL incase
of overflow |
-| `try_mul` | Multiplies operands (IntegerTypes only) or results NULL incase
of overflow |
-| `try_div` | Subtracts operands (IntegerTypes only) or results NULL incase
of overflow |
+Expressions that are not Spark-compatible are disabled by default and can be
enabled by setting
+`spark.comet.expression.allowIncompatible=true`.
## Conditional Expressions
-| Expression | Notes |
-| ---------- | ----- |
-| CaseWhen | |
-| If | |
-
-## Comparison
-
-| Expression | Notes |
-| ------------------------- | ----- |
-| EqualTo (`=`) | |
-| EqualNullSafe (`<=>`) | |
-| GreaterThan (`>`) | |
-| GreaterThanOrEqual (`>=`) | |
-| LessThan (`<`) | |
-| LessThanOrEqual (`<=`) | |
-| IsNull (`IS NULL`) | |
-| IsNotNull (`IS NOT NULL`) | |
-| In (`IN`) | |
+| Expression | SQL | Spark-Compatible?
|
+| ---------- | ------------------------------------------- | -----------------
|
+| CaseWhen | `CASE WHEN expr THEN expr ELSE expr END` | Yes
|
+| If | `IF(predicate_expr, true_expr, false_expr)` | Yes
|
+
+## Predicate Expressions
+
+| Expression | SQL | Spark-Compatible? |
+| ------------------ | ------------- | ----------------- |
+| And | `AND` | Yes |
+| EqualTo | `=` | Yes |
+| EqualNullSafe | `<=>` | Yes |
+| GreaterThan | `>` | Yes |
+| GreaterThanOrEqual | `>=` | Yes |
+| LessThan | `<` | Yes |
+| LessThanOrEqual | `<=` | Yes |
+| In | `IN` | Yes |
+| IsNotNull | `IS NOT NULL` | Yes |
+| IsNull | `IS NULL` | Yes |
+| InSet | `IN (...)` | Yes |
+| Not | `NOT` | Yes |
+| Or | `OR` | Yes |
## String Functions
-| Expression | Notes
|
-|-----------------|
-----------------------------------------------------------------------------------------------------------
|
-| Ascii |
|
-| BitLength |
|
-| Chr |
|
-| ConcatWs |
|
-| Contains |
|
-| EndsWith |
|
-| InitCap |
|
-| Instr |
|
-| Length |
|
-| Like |
|
-| Lower |
|
-| OctetLength |
|
-| Repeat | Negative argument for number of times to repeat causes
exception |
-| Replace |
|
-| Reverse |
|
-| StartsWith |
|
-| StringRPad |
|
-| StringSpace |
|
-| StringTrim |
|
-| StringTrimBoth |
|
-| StringTrimLeft |
|
-| StringTrimRight |
|
-| Substring |
|
-| Translate |
|
-| Upper |
|
+| Expression | Spark-Compatible? | Compatibility Notes
|
+| --------------- | ----------------- |
----------------------------------------------------------------------------------------------------------
|
+| Ascii | Yes |
|
+| BitLength | Yes |
|
+| Chr | Yes |
|
+| ConcatWs | Yes |
|
+| Contains | Yes |
|
+| EndsWith | Yes |
|
+| InitCap | No | Requires
`spark.comet.exec.initCap.enabled=true`
|
+| Length | Yes |
|
+| Like | Yes |
|
+| Lower | No | Results can vary depending on locale
and character set. Requires `spark.comet.caseConversion.enabled=true` |
+| OctetLength | Yes |
|
+| Reverse | Yes |
|
+| RLike | No | Uses Rust regexp engine, which has
different behavior to Java regexp engine |
+| StartsWith | Yes |
|
+| StringInstr | Yes |
|
+| StringRepeat | Yes | Negative argument for number of times
to repeat causes exception |
+| StringReplace | Yes |
|
+| StringRPad | Yes |
|
+| StringSpace | Yes |
|
+| StringTranslate | Yes |
|
+| StringTrim | Yes |
|
+| StringTrimBoth | Yes |
|
+| StringTrimLeft | Yes |
|
+| StringTrimRight | Yes |
|
+| Substring | Yes |
|
+| Upper | No | Results can vary depending on locale
and character set. Requires `spark.comet.caseConversion.enabled=true` |
## Date/Time Functions
-| Expression | Notes |
-| -------------- | ------------------------ |
-| DatePart | Only `year` is supported |
-| Extract | Only `year` is supported |
-| Hour | |
-| Minute | |
-| Second | |
-| TruncDate | |
-| TruncTimestamp | |
-| Year | |
+| Expression | SQL | Spark-Compatible? |
Compatibility Notes |
+| -------------- | ---------------------------- | ----------------- |
----------------------------------------------------------------------------- |
+| DateAdd | `date_add` | Yes |
|
+| DateSub | `date_sub` | Yes |
|
+| DatePart | `date_part(field, source)` | Yes | Only
`year` is supported |
+| Extract | `extract(field FROM source)` | Yes | Only
`year` is supported |
+| FromUnixTime | `from_unixtime` | No | Does not
support format, supports only -8334601211038 <= sec <= 8210266876799 |
+| Hour | `hour` | Yes |
|
+| Minute | `minute` | Yes |
|
+| Second | `second` | Yes |
|
+| TruncDate | `trunc` | Yes |
|
+| TruncTimestamp | `trunc_date` | Yes |
|
+| Year | `year` | Yes |
|
## Math Expressions
-| Expression | Notes
|
-| ---------- |
------------------------------------------------------------------- |
-| Abs |
|
-| Acos |
|
-| Asin |
|
-| Atan |
|
-| Atan2 |
|
-| Ceil |
|
-| Cos |
|
-| Exp |
|
-| Floor |
|
-| IsNaN |
|
-| Log |
|
-| Log2 |
|
-| Log10 |
|
-| Pow |
|
-| Round |
|
-| Signum | |
-| Sin |
|
-| Sqrt |
|
-| Tan |
|
+| Expression | SQL | Spark-Compatible? | Compatibility Notes
|
+| -------------- | --------- | ----------------- |
--------------------------------- |
+| Acos | `acos` | Yes |
|
+| Add | `+` | Yes | ANSI mode is not supported.
|
+| Asin | `asin` | Yes |
|
+| Atan | `atan` | Yes |
|
+| Atan2 | `atan2` | Yes |
|
+| BRound | `bround` | Yes | ANSI mode is not supported.
|
+| Ceil | `ceil` | Yes |
|
+| Cos | `cos` | Yes |
|
+| Divide | `/` | Yes | ANSI mode is not supported.
|
+| Exp | `exp` | Yes |
|
+| Expm1 | `expm1` | Yes |
|
+| Floor | `floor` | Yes |
|
+| Hex | `hex` | Yes |
|
+| IntegralDivide | `div` | Yes | ANSI mode is not supported.
|
+| IsNaN | `isnan` | Yes |
|
+| Log | `log` | Yes |
|
+| Log2 | `log2` | Yes |
|
+| Log10 | `log10` | Yes |
|
+| Multiply | `*` | Yes | ANSI mode is not supported.
|
+| Pow | `power` | Yes |
|
+| Rand | `rand` | Yes |
|
+| Randn | `randn` | Yes |
|
+| Remainder | `%` | Yes | ANSI mode is not supported.
|
+| Round | `round` | Yes | ANSI mode is not supported.
|
+| Signum | `signum` | Yes |
|
+| Sin | `sin` | Yes |
|
+| Sqrt | `sqrt` | Yes |
|
+| Subtract | `-` | Yes | ANSI mode is not supported.
|
+| Tan | `tan` | Yes |
|
+| TryAdd | `try_add` | Yes | Only integer inputs are
supported |
+| TryDivide | `try_div` | Yes | Only integer inputs are
supported |
+| TryMultiply | `try_mul` | Yes | Only integer inputs are
supported |
+| TrySubtract | `try_sub` | Yes | Only integer inputs are
supported |
+| UnaryMinus | `-` | Yes |
|
+| Unhex | `unhex` | Yes |
|
## Hashing Functions
-| Expression | Notes |
-| ---------- | ----- |
-| Md5 | |
-| Hash | |
-| Sha2 | |
-| XxHash64 | |
-
-## Boolean Expressions
-
-| Expression | Notes |
-| ---------- | ----- |
-| And | |
-| Or | |
-| Not | |
+| Expression | Spark-Compatible? |
+| ----------- | ----------------- |
+| Md5 | Yes |
+| Murmur3Hash | Yes |
+| Sha2 | Yes |
+| XxHash64 | Yes |
## Bitwise Expressions
-| Expression | Notes |
-| -------------------- | ----- |
-| ShiftLeft (`<<`) | |
-| ShiftRight (`>>`) | |
-| BitAnd (`&`) | |
-| BitOr (`\|`) | |
-| BitXor (`^`) | |
-| BitwiseNot (`~`) | |
-| BoolAnd (`bool_and`) | |
-| BoolOr (`bool_or`) | |
+| Expression | SQL | Spark-Compatible? |
+| ------------ | ---- | ----------------- |
+| BitwiseAnd | `&` | Yes |
+| BitwiseCount | | Yes |
+| BitwiseGet | | Yes |
+| BitwiseOr | `\|` | Yes |
+| BitwiseNot | `~` | Yes |
+| BitwiseXor | `^` | Yes |
+| ShiftLeft | `<<` | Yes |
+| ShiftRight | `>>` | Yes |
## Aggregate Expressions
-| Expression | Notes |
-| ------------- | ----- |
-| Avg | |
-| BitAndAgg | |
-| BitOrAgg | |
-| BitXorAgg | |
-| Corr | |
-| Count | |
-| CovPopulation | |
-| CovSample | |
-| First | |
-| Last | |
-| Max | |
-| Min | |
-| StddevPop | |
-| StddevSamp | |
-| Sum | |
-| VariancePop | |
-| VarianceSamp | |
-
-## Arrays
-
-| Expression | Notes
|
-|----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| ArrayAppend | Experimental
|
-| ArrayCompact | Experimental
|
-| ArrayContains | Experimental
|
-| ArrayDistinct | Experimental: behaves differently than spark. Datafusion
first sorts then removes duplicates while spark preserves the original order.
|
-| ArrayExcept | Experimental
|
-| ArrayInsert | Experimental
|
-| ArrayIntersect | Experimental
|
-| ArrayJoin | Experimental
|
-| ArrayMax | Experimental
|
-| ArrayMin |
|
-| ArrayRemove |
|
-| ArrayRepeat | Experimental
|
-| ArraysOverlap | Experimental
|
-| ArrayUnion | Experimental: behaves differently than spark. Datafusion
sorts the input arrays before performing the union, while spark preserves the
order of the first array and appends unique elements from the second. |
-| ElementAt | Arrays only
|
-| GetArrayItem |
-
-## Maps
-
-| Expression | Notes |
-|---------------------|--------------|
-| MapLookupByKey ([]) | |
-| MapKeys | |
-| MapValues | |
-| MapEntries | |
-| MapFromArrays | |
-
-## Structs
-
-| Expression | Notes |
-|-------------------|--------------|
-| CreateNamedStruct | |
-| GetStructField | |
-| StructsToJson | |
+| Expression | SQL | Spark-Compatible? | Compatibility Notes
|
+| ------------- | ---------- | ------------------------- |
---------------------------------------------------------------- |
+| Average | | Yes, except for ANSI mode |
|
+| BitAndAgg | | Yes |
|
+| BitOrAgg | | Yes |
|
+| BitXorAgg | | Yes |
|
+| BoolAnd | `bool_and` | Yes |
|
+| BoolOr | `bool_or` | Yes |
|
+| Corr | | Yes |
|
+| Count | | Yes |
|
+| CovPopulation | | Yes |
|
+| CovSample | | Yes |
|
+| First | | No | This function is
not deterministic. Results may not match Spark. |
+| Last | | No | This function is
not deterministic. Results may not match Spark. |
+| Max | | Yes |
|
+| Min | | Yes |
|
+| StddevPop | | Yes |
|
+| StddevSamp | | Yes |
|
+| Sum | | Yes, except for ANSI mode |
|
+| VariancePop | | Yes |
|
+| VarianceSamp | | Yes |
|
+
+## Array Expressions
+
+| Expression | Spark-Compatible? | Compatibility Notes
|
+| -------------- | ----------------- |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| ArrayAppend | No |
|
+| ArrayCompact | No |
|
+| ArrayContains | Yes |
|
+| ArrayDistinct | No | Behaves differently than spark. Comet
first sorts then removes duplicates while Spark preserves the original order.
|
+| ArrayExcept | No |
|
+| ArrayInsert | No |
|
+| ArrayIntersect | No |
|
+| ArrayJoin | No |
|
+| ArrayMax | Yes |
|
+| ArrayMin | Yes |
|
+| ArrayRemove | Yes |
|
+| ArrayRepeat | No |
|
+| ArrayUnion | No | Behaves differently than spark. Comet
sorts the input arrays before performing the union, while Spark preserves the
order of the first array and appends unique elements from the second. |
+| ArraysOverlap | No |
|
+| CreateArray | Yes |
|
+| ElementAt | Yes | Input must be an array. Map inputs are
not supported.
|
+| Flatten | Yes |
|
+| GetArrayItem | Yes |
|
+
+## Map Expressions
+
+| Expression | Spark-Compatible? |
+| ------------- | ----------------- |
+| GetMapValue | Yes |
+| MapKeys | Yes |
+| MapEntries | Yes |
+| MapValues | Yes |
+| MapFromArrays | Yes |
+
+## Struct Expressions
+
+| Expression | Spark-Compatible? |
+| -------------------- | ----------------- |
+| CreateNamedStruct | Yes |
+| GetArrayStructFields | Yes |
+| GetStructField | Yes |
+| StructsToJson | Yes |
+
+## Conversion Expressions
+
+| Expression | Spark-Compatible | Compatibility Notes
|
+| ---------- | ------------------------ |
-------------------------------------------------------------------------------------------
|
+| Cast | Depends on specific cast | See the [Comet Compatibility Guide]
for list of supported cast expressions and known issues |
## Other
-| Expression | Notes
|
-|-------------------------|
-------------------------------------------------------------------------------
|
-| Cast | See compatibility guide for list of supported cast
expressions and known issues |
-| BloomFilterMightContain |
|
-| ScalarSubquery |
|
-| Coalesce |
|
-| NormalizeNaNAndZero |
|
-| ToPrettyString |
|
-| FromUnixTime | Does not support format, supports only
-8334601211038 <= sec <= 8210266876799 |
\ No newline at end of file
+| Expression | Spark-Compatible? | Compatibility Notes
|
+| ---------------------------- | ----------------- |
--------------------------------------------------------------------------- |
+| Alias | Yes |
|
+| AttributeRefernce | Yes |
|
+| BloomFilterMightContain | Yes |
|
+| Coalesce | Yes |
|
+| CheckOverflow | Yes |
|
+| KnownFloatingPointNormalized | Yes |
|
+| Literal | Yes |
|
+| MakeDecimal | Yes |
|
+| MonotonicallyIncreasingID | Yes |
|
+| NormalizeNaNAndZero | Yes |
|
+| PromotePrecision | Yes |
|
+| RegExpReplace | No | Uses Rust regexp engine,
which has different behavior to Java regexp engine |
+| ScalarSubquery | Yes |
|
+| SparkPartitionID | Yes |
|
+| ToPrettyString | Yes |
|
+| UnscaledValue | Yes |
|
+
+[Comet Compatibility Guide]: compatibility.md
diff --git a/_sources/user-guide/latest/operators.md.txt
b/_sources/user-guide/latest/operators.md.txt
index 7edbb1e70..066daf4b9 100644
--- a/_sources/user-guide/latest/operators.md.txt
+++ b/_sources/user-guide/latest/operators.md.txt
@@ -22,16 +22,24 @@
The following Spark operators are currently replaced with native versions.
Query stages that contain any operators
not supported by Comet will fall back to regular Spark execution.
-| Operator | Notes |
-| --------------------- | ----- |
-| Projection | |
-| Filter | |
-| Sort | |
-| Hash Aggregate | |
-| Limit | |
-| Sort-merge Join | |
-| Hash Join | |
-| BroadcastHashJoinExec | |
-| Shuffle | |
-| Expand | |
-| Union | |
+| Operator | Spark-Compatible? | Compatibility Notes
|
+| ----------------------- | ----------------- |
------------------------------------------------------------------------------------------------------------------
|
+| BatchScanExec | Yes | Supports Parquet files and
Apache Iceberg Parquet scans. See the [Comet Compatibility Guide] for more
information. |
+| BroadcastExchangeExec | Yes |
|
+| BroadcastHashJoinExec | Yes |
|
+| ExpandExec | Yes |
|
+| FileSourceScanExec | Yes | Supports Parquet files. See
the [Comet Compatibility Guide] for more information.
|
+| FilterExec | Yes |
|
+| GlobalLimitExec | Yes |
|
+| HashAggregateExec | Yes |
|
+| LocalLimitExec | Yes |
|
+| ObjectHashAggregateExec | Yes | Supports a limited number of
aggregates, such as `bloom_filter_agg`.
|
+| ProjectExec | Yes |
|
+| ShuffleExchangeExec | Yes |
|
+| ShuffledHashJoinExec | Yes |
|
+| SortExec | Yes |
|
+| SortMergeJoinExec | Yes |
|
+| UnionExec | Yes |
|
+| WindowExec | Yes |
|
+
+[Comet Compatibility Guide]: compatibility.md
diff --git a/searchindex.js b/searchindex.js
index 60e14932c..7942b43db 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[12, "install-comet"]],
"2. Clone Spark and Apply Diff": [[12, "clone-spark-and-apply-diff"]], "3. Run
Spark SQL Tests": [[12, "run-spark-sql-tests"]], "ANSI Mode": [[43,
"ansi-mode"]], "ANSI mode": [[17, "ansi-mode"], [30, "ansi-mode"]], "API
Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "Accelerating Apache Iceberg
Parquet Scans using Comet (Experimental)": [[22, null], [35, null], [48,
null]], [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[12, "install-comet"]],
"2. Clone Spark and Apply Diff": [[12, "clone-spark-and-apply-diff"]], "3. Run
Spark SQL Tests": [[12, "run-spark-sql-tests"]], "ANSI Mode": [[43,
"ansi-mode"]], "ANSI mode": [[17, "ansi-mode"], [30, "ansi-mode"]], "API
Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "Accelerating Apache Iceberg
Parquet Scans using Comet (Experimental)": [[22, null], [35, null], [48,
null]], [...]
\ No newline at end of file
diff --git a/user-guide/latest/datatypes.html b/user-guide/latest/datatypes.html
index ef4962db7..d45276871 100644
--- a/user-guide/latest/datatypes.html
+++ b/user-guide/latest/datatypes.html
@@ -512,39 +512,51 @@ under the License.
-->
<section id="supported-spark-data-types">
<h1>Supported Spark Data Types<a class="headerlink"
href="#supported-spark-data-types" title="Link to this heading">¶</a></h1>
-<p>The following Spark data types are currently available:</p>
+<p>Comet supports the following Spark data types. Refer to the <a
class="reference internal" href="compatibility.html"><span class="std
std-doc">Comet Compatibility Guide</span></a> for information about data
+type support in scans and other operators.</p>
<!-- based on org.apache.comet.serde.QueryPlanSerde.supportedDataType -->
-<ul class="simple">
-<li><p>Primitives</p>
-<ul>
-<li><p>Boolean</p></li>
-<li><p>Byte</p></li>
-<li><p>Short</p></li>
-<li><p>Integer</p></li>
-<li><p>Long</p></li>
-<li><p>Float</p></li>
-<li><p>Double</p></li>
-</ul>
-</li>
-<li><p>String</p></li>
-<li><p>Binary</p></li>
-<li><p>Decimal</p></li>
-<li><p>Temporal</p>
-<ul>
-<li><p>Date</p></li>
-<li><p>Timestamp</p></li>
-<li><p>TimestampNTZ</p></li>
-</ul>
-</li>
-<li><p>Null</p></li>
-<li><p>Complex Types</p>
-<ul>
-<li><p>Struct</p></li>
-<li><p>Array</p></li>
-<li><p>Map</p></li>
-</ul>
-</li>
-</ul>
+<table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Data Type</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p>Null</p></td>
+</tr>
+<tr class="row-odd"><td><p>Boolean</p></td>
+</tr>
+<tr class="row-even"><td><p>Byte</p></td>
+</tr>
+<tr class="row-odd"><td><p>Short</p></td>
+</tr>
+<tr class="row-even"><td><p>Integer</p></td>
+</tr>
+<tr class="row-odd"><td><p>Long</p></td>
+</tr>
+<tr class="row-even"><td><p>Float</p></td>
+</tr>
+<tr class="row-odd"><td><p>Double</p></td>
+</tr>
+<tr class="row-even"><td><p>Decimal</p></td>
+</tr>
+<tr class="row-odd"><td><p>String</p></td>
+</tr>
+<tr class="row-even"><td><p>Binary</p></td>
+</tr>
+<tr class="row-odd"><td><p>Date</p></td>
+</tr>
+<tr class="row-even"><td><p>Timestamp</p></td>
+</tr>
+<tr class="row-odd"><td><p>TimestampNTZ</p></td>
+</tr>
+<tr class="row-even"><td><p>Struct</p></td>
+</tr>
+<tr class="row-odd"><td><p>Array</p></td>
+</tr>
+<tr class="row-even"><td><p>Map</p></td>
+</tr>
+</tbody>
+</table>
</section>
diff --git a/user-guide/latest/expressions.html
b/user-guide/latest/expressions.html
index c2d26a31e..83d23b3aa 100644
--- a/user-guide/latest/expressions.html
+++ b/user-guide/latest/expressions.html
@@ -469,34 +469,14 @@ under the License.
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#literal-values">
- Literal Values
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#unary-arithmetic">
- Unary Arithmetic
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#binary-arithmetic">
- Binary Arithmetic
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#binary-try-arithmetic">
- Binary Try Arithmetic
- </a>
- </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#conditional-expressions">
Conditional Expressions
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#comparison">
- Comparison
+ <a class="reference internal nav-link" href="#predicate-expressions">
+ Predicate Expressions
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -519,11 +499,6 @@ under the License.
Hashing Functions
</a>
</li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#boolean-expressions">
- Boolean Expressions
- </a>
- </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#bitwise-expressions">
Bitwise Expressions
@@ -535,18 +510,23 @@ under the License.
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#arrays">
- Arrays
+ <a class="reference internal nav-link" href="#array-expressions">
+ Array Expressions
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#maps">
- Maps
+ <a class="reference internal nav-link" href="#map-expressions">
+ Map Expressions
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#structs">
- Structs
+ <a class="reference internal nav-link" href="#struct-expressions">
+ Struct Expressions
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#conversion-expressions">
+ Conversion Expressions
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -602,144 +582,93 @@ under the License.
-->
<section id="supported-spark-expressions">
<h1>Supported Spark Expressions<a class="headerlink"
href="#supported-spark-expressions" title="Link to this heading">¶</a></h1>
-<p>The following Spark expressions are currently available. Any known
compatibility issues are noted in the following tables.</p>
-<section id="literal-values">
-<h2>Literal Values<a class="headerlink" href="#literal-values" title="Link to
this heading">¶</a></h2>
-<table class="table">
-<thead>
-<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
-</tr>
-</thead>
-<tbody>
-<tr class="row-even"><td><p>Literal values of supported data types</p></td>
-<td><p></p></td>
-</tr>
-</tbody>
-</table>
-</section>
-<section id="unary-arithmetic">
-<h2>Unary Arithmetic<a class="headerlink" href="#unary-arithmetic" title="Link
to this heading">¶</a></h2>
-<table class="table">
-<thead>
-<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
-</tr>
-</thead>
-<tbody>
-<tr class="row-even"><td><p>UnaryMinus (<code class="docutils literal
notranslate"><span class="pre">-</span></code>)</p></td>
-<td><p></p></td>
-</tr>
-</tbody>
-</table>
-</section>
-<section id="binary-arithmetic">
-<h2>Binary Arithmetic<a class="headerlink" href="#binary-arithmetic"
title="Link to this heading">¶</a></h2>
-<table class="table">
-<thead>
-<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
-</tr>
-</thead>
-<tbody>
-<tr class="row-even"><td><p>Add (<code class="docutils literal
notranslate"><span class="pre">+</span></code>)</p></td>
-<td><p></p></td>
-</tr>
-<tr class="row-odd"><td><p>Subtract (<code class="docutils literal
notranslate"><span class="pre">-</span></code>)</p></td>
-<td><p></p></td>
-</tr>
-<tr class="row-even"><td><p>Multiply (<code class="docutils literal
notranslate"><span class="pre">*</span></code>)</p></td>
-<td><p></p></td>
-</tr>
-<tr class="row-odd"><td><p>Divide (<code class="docutils literal
notranslate"><span class="pre">/</span></code>)</p></td>
-<td><p></p></td>
-</tr>
-<tr class="row-even"><td><p>IntegralDivide (<code class="docutils literal
notranslate"><span class="pre">div</span></code>)</p></td>
-<td><p>All operands are cast to DecimalType (in case the input type is not
already decima type) with precision 19 and scale 0. Please set <code
class="docutils literal notranslate"><span
class="pre">spark.comet.expression.allowIncompatible</span></code> to <code
class="docutils literal notranslate"><span class="pre">true</span></code> to
enable DataFusion’s cast operation for LongType inputs.</p></td>
-</tr>
-<tr class="row-odd"><td><p>Remainder (<code class="docutils literal
notranslate"><span class="pre">%</span></code>)</p></td>
-<td><p></p></td>
-</tr>
-</tbody>
-</table>
-</section>
-<section id="binary-try-arithmetic">
-<h2>Binary Try Arithmetic<a class="headerlink" href="#binary-try-arithmetic"
title="Link to this heading">¶</a></h2>
-<table class="table">
-<thead>
-<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
-</tr>
-</thead>
-<tbody>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">try_add</span></code></p></td>
-<td><p>Adds operands (IntegerTypes only) or results NULL incase of
overflow</p></td>
-</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">try_sub</span></code></p></td>
-<td><p>Subtracts operands (IntegerTypes only) or results NULL incase of
overflow</p></td>
-</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">try_mul</span></code></p></td>
-<td><p>Multiplies operands (IntegerTypes only) or results NULL incase of
overflow</p></td>
-</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">try_div</span></code></p></td>
-<td><p>Subtracts operands (IntegerTypes only) or results NULL incase of
overflow</p></td>
-</tr>
-</tbody>
-</table>
-</section>
+<p>Comet supports the following Spark expressions. Expressions that are marked
as Spark-compatible will either run
+natively in Comet and provide the same results as Spark, or will fall back to
Spark for cases that would not
+be compatible.</p>
+<p>Expressions that are not Spark-compatible are disabled by default and can
be enabled by setting
+<code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.allowIncompatible=true</span></code>.</p>
<section id="conditional-expressions">
<h2>Conditional Expressions<a class="headerlink"
href="#conditional-expressions" title="Link to this heading">¶</a></h2>
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>SQL</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>CaseWhen</p></td>
-<td><p></p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">CASE</span> <span class="pre">WHEN</span> <span
class="pre">expr</span> <span class="pre">THEN</span> <span
class="pre">expr</span> <span class="pre">ELSE</span> <span
class="pre">expr</span> <span class="pre">END</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td><p>If</p></td>
-<td><p></p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">IF(predicate_expr,</span> <span class="pre">true_expr,</span> <span
class="pre">false_expr)</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
</tbody>
</table>
</section>
-<section id="comparison">
-<h2>Comparison<a class="headerlink" href="#comparison" title="Link to this
heading">¶</a></h2>
+<section id="predicate-expressions">
+<h2>Predicate Expressions<a class="headerlink" href="#predicate-expressions"
title="Link to this heading">¶</a></h2>
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>SQL</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>EqualTo (<code class="docutils literal
notranslate"><span class="pre">=</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>And</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">AND</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-odd"><td><p>EqualNullSafe (<code class="docutils literal
notranslate"><span class="pre"><=></span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-odd"><td><p>EqualTo</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">=</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-even"><td><p>GreaterThan (<code class="docutils literal
notranslate"><span class="pre">></span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>EqualNullSafe</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre"><=></span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-odd"><td><p>GreaterThanOrEqual (<code class="docutils literal
notranslate"><span class="pre">>=</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-odd"><td><p>GreaterThan</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">></span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-even"><td><p>LessThan (<code class="docutils literal
notranslate"><span class="pre"><</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>GreaterThanOrEqual</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">>=</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-odd"><td><p>LessThanOrEqual (<code class="docutils literal
notranslate"><span class="pre"><=</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-odd"><td><p>LessThan</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre"><</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-even"><td><p>IsNull (<code class="docutils literal
notranslate"><span class="pre">IS</span> <span
class="pre">NULL</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>LessThanOrEqual</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre"><=</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-odd"><td><p>IsNotNull (<code class="docutils literal
notranslate"><span class="pre">IS</span> <span class="pre">NOT</span> <span
class="pre">NULL</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-odd"><td><p>In</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">IN</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-even"><td><p>In (<code class="docutils literal
notranslate"><span class="pre">IN</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>IsNotNull</p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">IS</span>
<span class="pre">NOT</span> <span class="pre">NULL</span></code></p></td>
+<td><p>Yes</p></td>
+</tr>
+<tr class="row-odd"><td><p>IsNull</p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">IS</span>
<span class="pre">NULL</span></code></p></td>
+<td><p>Yes</p></td>
+</tr>
+<tr class="row-even"><td><p>InSet</p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">IN</span>
<span class="pre">(...)</span></code></p></td>
+<td><p>Yes</p></td>
+</tr>
+<tr class="row-odd"><td><p>Not</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">NOT</span></code></p></td>
+<td><p>Yes</p></td>
+</tr>
+<tr class="row-even"><td><p>Or</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">OR</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
</tbody>
</table>
@@ -749,84 +678,114 @@ under the License.
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
+<th class="head"><p>Compatibility Notes</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>Ascii</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>BitLength</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-even"><td><p>Chr</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>ConcatWs</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-even"><td><p>Contains</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>EndsWith</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-even"><td><p>InitCap</p></td>
+<td><p>No</p></td>
+<td><p>Requires <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.initCap.enabled=true</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p>Length</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Instr</p></td>
+<tr class="row-even"><td><p>Like</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Length</p></td>
+<tr class="row-odd"><td><p>Lower</p></td>
+<td><p>No</p></td>
+<td><p>Results can vary depending on locale and character set. Requires <code
class="docutils literal notranslate"><span
class="pre">spark.comet.caseConversion.enabled=true</span></code></p></td>
+</tr>
+<tr class="row-even"><td><p>OctetLength</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Like</p></td>
+<tr class="row-odd"><td><p>Reverse</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Lower</p></td>
+<tr class="row-even"><td><p>RLike</p></td>
+<td><p>No</p></td>
+<td><p>Uses Rust regexp engine, which has different behavior to Java regexp
engine</p></td>
+</tr>
+<tr class="row-odd"><td><p>StartsWith</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>OctetLength</p></td>
+<tr class="row-even"><td><p>StringInstr</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Repeat</p></td>
+<tr class="row-odd"><td><p>StringRepeat</p></td>
+<td><p>Yes</p></td>
<td><p>Negative argument for number of times to repeat causes
exception</p></td>
</tr>
-<tr class="row-odd"><td><p>Replace</p></td>
-<td><p></p></td>
-</tr>
-<tr class="row-even"><td><p>Reverse</p></td>
+<tr class="row-even"><td><p>StringReplace</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>StartsWith</p></td>
+<tr class="row-odd"><td><p>StringRPad</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>StringRPad</p></td>
+<tr class="row-even"><td><p>StringSpace</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>StringSpace</p></td>
+<tr class="row-odd"><td><p>StringTranslate</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-even"><td><p>StringTrim</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>StringTrimBoth</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-even"><td><p>StringTrimLeft</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>StringTrimRight</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-even"><td><p>Substring</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Translate</p></td>
-<td><p></p></td>
-</tr>
-<tr class="row-even"><td><p>Upper</p></td>
-<td><p></p></td>
+<tr class="row-odd"><td><p>Upper</p></td>
+<td><p>No</p></td>
+<td><p>Results can vary depending on locale and character set. Requires <code
class="docutils literal notranslate"><span
class="pre">spark.comet.caseConversion.enabled=true</span></code></p></td>
</tr>
</tbody>
</table>
@@ -836,32 +795,65 @@ under the License.
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>SQL</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
+<th class="head"><p>Compatibility Notes</p></th>
</tr>
</thead>
<tbody>
+<tr class="row-even"><td><p>DateAdd</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">date_add</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-odd"><td><p>DateSub</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">date_sub</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
<tr class="row-even"><td><p>DatePart</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">date_part(field,</span> <span
class="pre">source)</span></code></p></td>
+<td><p>Yes</p></td>
<td><p>Only <code class="docutils literal notranslate"><span
class="pre">year</span></code> is supported</p></td>
</tr>
<tr class="row-odd"><td><p>Extract</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">extract(field</span> <span class="pre">FROM</span> <span
class="pre">source)</span></code></p></td>
+<td><p>Yes</p></td>
<td><p>Only <code class="docutils literal notranslate"><span
class="pre">year</span></code> is supported</p></td>
</tr>
-<tr class="row-even"><td><p>Hour</p></td>
+<tr class="row-even"><td><p>FromUnixTime</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">from_unixtime</span></code></p></td>
+<td><p>No</p></td>
+<td><p>Does not support format, supports only -8334601211038 <= sec <=
8210266876799</p></td>
+</tr>
+<tr class="row-odd"><td><p>Hour</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">hour</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Minute</p></td>
+<tr class="row-even"><td><p>Minute</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">minute</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Second</p></td>
+<tr class="row-odd"><td><p>Second</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">second</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>TruncDate</p></td>
+<tr class="row-even"><td><p>TruncDate</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">trunc</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>TruncTimestamp</p></td>
+<tr class="row-odd"><td><p>TruncTimestamp</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">trunc_date</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Year</p></td>
+<tr class="row-even"><td><p>Year</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">year</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
</tbody>
@@ -872,111 +864,210 @@ under the License.
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>SQL</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
+<th class="head"><p>Compatibility Notes</p></th>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>Abs</p></td>
+<tr class="row-even"><td><p>Acos</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">acos</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Acos</p></td>
-<td><p></p></td>
+<tr class="row-odd"><td><p>Add</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">+</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p>ANSI mode is not supported.</p></td>
</tr>
<tr class="row-even"><td><p>Asin</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">asin</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>Atan</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">atan</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-even"><td><p>Atan2</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">atan2</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Ceil</p></td>
+<tr class="row-odd"><td><p>BRound</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">bround</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p>ANSI mode is not supported.</p></td>
+</tr>
+<tr class="row-even"><td><p>Ceil</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">ceil</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Cos</p></td>
+<tr class="row-odd"><td><p>Cos</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">cos</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
+<tr class="row-even"><td><p>Divide</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">/</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p>ANSI mode is not supported.</p></td>
+</tr>
<tr class="row-odd"><td><p>Exp</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">exp</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Floor</p></td>
+<tr class="row-even"><td><p>Expm1</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">expm1</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>IsNaN</p></td>
+<tr class="row-odd"><td><p>Floor</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">floor</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Log</p></td>
+<tr class="row-even"><td><p>Hex</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">hex</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Log2</p></td>
+<tr class="row-odd"><td><p>IntegralDivide</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">div</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p>ANSI mode is not supported.</p></td>
+</tr>
+<tr class="row-even"><td><p>IsNaN</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">isnan</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Log10</p></td>
+<tr class="row-odd"><td><p>Log</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">log</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Pow</p></td>
+<tr class="row-even"><td><p>Log2</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">log2</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Round</p></td>
+<tr class="row-odd"><td><p>Log10</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">log10</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Signum</p></td>
+<tr class="row-even"><td><p>Multiply</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">*</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p>ANSI mode is not supported.</p></td>
+</tr>
+<tr class="row-odd"><td><p>Pow</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">power</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Sin</p></td>
+<tr class="row-even"><td><p>Rand</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">rand</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Sqrt</p></td>
+<tr class="row-odd"><td><p>Randn</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">randn</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Tan</p></td>
+<tr class="row-even"><td><p>Remainder</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">%</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p>ANSI mode is not supported.</p></td>
+</tr>
+<tr class="row-odd"><td><p>Round</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">round</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p>ANSI mode is not supported.</p></td>
+</tr>
+<tr class="row-even"><td><p>Signum</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">signum</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-</tbody>
-</table>
-</section>
-<section id="hashing-functions">
-<h2>Hashing Functions<a class="headerlink" href="#hashing-functions"
title="Link to this heading">¶</a></h2>
-<table class="table">
-<thead>
-<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<tr class="row-odd"><td><p>Sin</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">sin</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
-</thead>
-<tbody>
-<tr class="row-even"><td><p>Md5</p></td>
+<tr class="row-even"><td><p>Sqrt</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">sqrt</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Hash</p></td>
+<tr class="row-odd"><td><p>Subtract</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">-</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p>ANSI mode is not supported.</p></td>
+</tr>
+<tr class="row-even"><td><p>Tan</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">tan</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Sha2</p></td>
+<tr class="row-odd"><td><p>TryAdd</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">try_add</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p>Only integer inputs are supported</p></td>
+</tr>
+<tr class="row-even"><td><p>TryDivide</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">try_div</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p>Only integer inputs are supported</p></td>
+</tr>
+<tr class="row-odd"><td><p>TryMultiply</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">try_mul</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p>Only integer inputs are supported</p></td>
+</tr>
+<tr class="row-even"><td><p>TrySubtract</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">try_sub</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p>Only integer inputs are supported</p></td>
+</tr>
+<tr class="row-odd"><td><p>UnaryMinus</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">-</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>XxHash64</p></td>
+<tr class="row-even"><td><p>Unhex</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">unhex</span></code></p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
</tbody>
</table>
</section>
-<section id="boolean-expressions">
-<h2>Boolean Expressions<a class="headerlink" href="#boolean-expressions"
title="Link to this heading">¶</a></h2>
+<section id="hashing-functions">
+<h2>Hashing Functions<a class="headerlink" href="#hashing-functions"
title="Link to this heading">¶</a></h2>
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>And</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>Md5</p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-odd"><td><p>Or</p></td>
-<td><p></p></td>
+<tr class="row-odd"><td><p>Murmur3Hash</p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-even"><td><p>Not</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>Sha2</p></td>
+<td><p>Yes</p></td>
+</tr>
+<tr class="row-odd"><td><p>XxHash64</p></td>
+<td><p>Yes</p></td>
</tr>
</tbody>
</table>
@@ -986,33 +1077,42 @@ under the License.
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>SQL</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>ShiftLeft (<code class="docutils literal
notranslate"><span class="pre"><<</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>BitwiseAnd</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">&</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-odd"><td><p>ShiftRight (<code class="docutils literal
notranslate"><span class="pre">>></span></code>)</p></td>
+<tr class="row-odd"><td><p>BitwiseCount</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-even"><td><p>BitAnd (<code class="docutils literal
notranslate"><span class="pre">&</span></code>)</p></td>
+<tr class="row-even"><td><p>BitwiseGet</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-odd"><td><p>BitOr (<code class="docutils literal
notranslate"><span class="pre">|</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-odd"><td><p>BitwiseOr</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">|</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-even"><td><p>BitXor (<code class="docutils literal
notranslate"><span class="pre">^</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>BitwiseNot</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">~</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-odd"><td><p>BitwiseNot (<code class="docutils literal
notranslate"><span class="pre">~</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-odd"><td><p>BitwiseXor</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">^</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-even"><td><p>BoolAnd (<code class="docutils literal
notranslate"><span class="pre">bool_and</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>ShiftLeft</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre"><<</span></code></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-odd"><td><p>BoolOr (<code class="docutils literal
notranslate"><span class="pre">bool_or</span></code>)</p></td>
-<td><p></p></td>
+<tr class="row-odd"><td><p>ShiftRight</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">>></span></code></p></td>
+<td><p>Yes</p></td>
</tr>
</tbody>
</table>
@@ -1022,168 +1122,259 @@ under the License.
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>SQL</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
+<th class="head"><p>Compatibility Notes</p></th>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>Avg</p></td>
+<tr class="row-even"><td><p>Average</p></td>
+<td><p></p></td>
+<td><p>Yes, except for ANSI mode</p></td>
<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>BitAndAgg</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p>BitOrAgg</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>BitXorAgg</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-even"><td><p>BoolAnd</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">bool_and</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-odd"><td><p>BoolOr</p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">bool_or</span></code></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p>Corr</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>Count</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p>CovPopulation</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>CovSample</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p>First</p></td>
<td><p></p></td>
+<td><p>No</p></td>
+<td><p>This function is not deterministic. Results may not match
Spark.</p></td>
</tr>
<tr class="row-odd"><td><p>Last</p></td>
<td><p></p></td>
+<td><p>No</p></td>
+<td><p>This function is not deterministic. Results may not match
Spark.</p></td>
</tr>
<tr class="row-even"><td><p>Max</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>Min</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p>StddevPop</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>StddevSamp</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p>Sum</p></td>
<td><p></p></td>
+<td><p>Yes, except for ANSI mode</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>VariancePop</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p>VarianceSamp</p></td>
<td><p></p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
</tbody>
</table>
</section>
-<section id="arrays">
-<h2>Arrays<a class="headerlink" href="#arrays" title="Link to this
heading">¶</a></h2>
+<section id="array-expressions">
+<h2>Array Expressions<a class="headerlink" href="#array-expressions"
title="Link to this heading">¶</a></h2>
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
+<th class="head"><p>Compatibility Notes</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>ArrayAppend</p></td>
-<td><p>Experimental</p></td>
+<td><p>No</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>ArrayCompact</p></td>
-<td><p>Experimental</p></td>
+<td><p>No</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p>ArrayContains</p></td>
-<td><p>Experimental</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>ArrayDistinct</p></td>
-<td><p>Experimental: behaves differently than spark. Datafusion first sorts
then removes duplicates while spark preserves the original order.</p></td>
+<td><p>No</p></td>
+<td><p>Behaves differently than spark. Comet first sorts then removes
duplicates while Spark preserves the original order.</p></td>
</tr>
<tr class="row-even"><td><p>ArrayExcept</p></td>
-<td><p>Experimental</p></td>
+<td><p>No</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>ArrayInsert</p></td>
-<td><p>Experimental</p></td>
+<td><p>No</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p>ArrayIntersect</p></td>
-<td><p>Experimental</p></td>
+<td><p>No</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>ArrayJoin</p></td>
-<td><p>Experimental</p></td>
+<td><p>No</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p>ArrayMax</p></td>
-<td><p>Experimental</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>ArrayMin</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-even"><td><p>ArrayRemove</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>ArrayRepeat</p></td>
-<td><p>Experimental</p></td>
+<td><p>No</p></td>
+<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>ArraysOverlap</p></td>
-<td><p>Experimental</p></td>
+<tr class="row-even"><td><p>ArrayUnion</p></td>
+<td><p>No</p></td>
+<td><p>Behaves differently than spark. Comet sorts the input arrays before
performing the union, while Spark preserves the order of the first array and
appends unique elements from the second.</p></td>
</tr>
-<tr class="row-odd"><td><p>ArrayUnion</p></td>
-<td><p>Experimental: behaves differently than spark. Datafusion sorts the
input arrays before performing the union, while spark preserves the order of
the first array and appends unique elements from the second.</p></td>
+<tr class="row-odd"><td><p>ArraysOverlap</p></td>
+<td><p>No</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-even"><td><p>CreateArray</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>ElementAt</p></td>
-<td><p>Arrays only</p></td>
+<tr class="row-odd"><td><p>ElementAt</p></td>
+<td><p>Yes</p></td>
+<td><p>Input must be an array. Map inputs are not supported.</p></td>
+</tr>
+<tr class="row-even"><td><p>Flatten</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>GetArrayItem</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
</tbody>
</table>
</section>
-<section id="maps">
-<h2>Maps<a class="headerlink" href="#maps" title="Link to this
heading">¶</a></h2>
+<section id="map-expressions">
+<h2>Map Expressions<a class="headerlink" href="#map-expressions" title="Link
to this heading">¶</a></h2>
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>MapLookupByKey ([])</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>GetMapValue</p></td>
+<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td><p>MapKeys</p></td>
-<td><p></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-even"><td><p>MapValues</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>MapEntries</p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-odd"><td><p>MapEntries</p></td>
-<td><p></p></td>
+<tr class="row-odd"><td><p>MapValues</p></td>
+<td><p>Yes</p></td>
</tr>
<tr class="row-even"><td><p>MapFromArrays</p></td>
-<td><p></p></td>
+<td><p>Yes</p></td>
</tr>
</tbody>
</table>
</section>
-<section id="structs">
-<h2>Structs<a class="headerlink" href="#structs" title="Link to this
heading">¶</a></h2>
+<section id="struct-expressions">
+<h2>Struct Expressions<a class="headerlink" href="#struct-expressions"
title="Link to this heading">¶</a></h2>
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>CreateNamedStruct</p></td>
-<td><p></p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-odd"><td><p>GetStructField</p></td>
-<td><p></p></td>
+<tr class="row-odd"><td><p>GetArrayStructFields</p></td>
+<td><p>Yes</p></td>
</tr>
-<tr class="row-even"><td><p>StructsToJson</p></td>
-<td><p></p></td>
+<tr class="row-even"><td><p>GetStructField</p></td>
+<td><p>Yes</p></td>
+</tr>
+<tr class="row-odd"><td><p>StructsToJson</p></td>
+<td><p>Yes</p></td>
+</tr>
+</tbody>
+</table>
+</section>
+<section id="conversion-expressions">
+<h2>Conversion Expressions<a class="headerlink" href="#conversion-expressions"
title="Link to this heading">¶</a></h2>
+<table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Expression</p></th>
+<th class="head"><p>Spark-Compatible</p></th>
+<th class="head"><p>Compatibility Notes</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p>Cast</p></td>
+<td><p>Depends on specific cast</p></td>
+<td><p>See the <a class="reference internal" href="compatibility.html"><span
class="std std-doc">Comet Compatibility Guide</span></a> for list of supported
cast expressions and known issues</p></td>
</tr>
</tbody>
</table>
@@ -1193,30 +1384,74 @@ under the License.
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Expression</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
+<th class="head"><p>Compatibility Notes</p></th>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>Cast</p></td>
-<td><p>See compatibility guide for list of supported cast expressions and
known issues</p></td>
+<tr class="row-even"><td><p>Alias</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>BloomFilterMightContain</p></td>
+<tr class="row-odd"><td><p>AttributeRefernce</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>ScalarSubquery</p></td>
+<tr class="row-even"><td><p>BloomFilterMightContain</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>Coalesce</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>NormalizeNaNAndZero</p></td>
+<tr class="row-even"><td><p>CheckOverflow</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>ToPrettyString</p></td>
+<tr class="row-odd"><td><p>KnownFloatingPointNormalized</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>FromUnixTime</p></td>
-<td><p>Does not support format, supports only -8334601211038 <= sec <=
8210266876799</p></td>
+<tr class="row-even"><td><p>Literal</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-odd"><td><p>MakeDecimal</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-even"><td><p>MonotonicallyIncreasingID</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-odd"><td><p>NormalizeNaNAndZero</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-even"><td><p>PromotePrecision</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-odd"><td><p>RegExpReplace</p></td>
+<td><p>No</p></td>
+<td><p>Uses Rust regexp engine, which has different behavior to Java regexp
engine</p></td>
+</tr>
+<tr class="row-even"><td><p>ScalarSubquery</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-odd"><td><p>SparkPartitionID</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-even"><td><p>ToPrettyString</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-odd"><td><p>UnscaledValue</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
</tr>
</tbody>
</table>
diff --git a/user-guide/latest/operators.html b/user-guide/latest/operators.html
index 6e7712bef..21a567097 100644
--- a/user-guide/latest/operators.html
+++ b/user-guide/latest/operators.html
@@ -517,41 +517,77 @@ not supported by Comet will fall back to regular Spark
execution.</p>
<table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Operator</p></th>
-<th class="head"><p>Notes</p></th>
+<th class="head"><p>Spark-Compatible?</p></th>
+<th class="head"><p>Compatibility Notes</p></th>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>Projection</p></td>
+<tr class="row-even"><td><p>BatchScanExec</p></td>
+<td><p>Yes</p></td>
+<td><p>Supports Parquet files and Apache Iceberg Parquet scans. See the <a
class="reference internal" href="compatibility.html"><span class="std
std-doc">Comet Compatibility Guide</span></a> for more information.</p></td>
+</tr>
+<tr class="row-odd"><td><p>BroadcastExchangeExec</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Filter</p></td>
+<tr class="row-even"><td><p>BroadcastHashJoinExec</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Sort</p></td>
+<tr class="row-odd"><td><p>ExpandExec</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Hash Aggregate</p></td>
+<tr class="row-even"><td><p>FileSourceScanExec</p></td>
+<td><p>Yes</p></td>
+<td><p>Supports Parquet files. See the <a class="reference internal"
href="compatibility.html"><span class="std std-doc">Comet Compatibility
Guide</span></a> for more information.</p></td>
+</tr>
+<tr class="row-odd"><td><p>FilterExec</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Limit</p></td>
+<tr class="row-even"><td><p>GlobalLimitExec</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-odd"><td><p>HashAggregateExec</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-even"><td><p>LocalLimitExec</p></td>
+<td><p>Yes</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-odd"><td><p>ObjectHashAggregateExec</p></td>
+<td><p>Yes</p></td>
+<td><p>Supports a limited number of aggregates, such as <code class="docutils
literal notranslate"><span class="pre">bloom_filter_agg</span></code>.</p></td>
+</tr>
+<tr class="row-even"><td><p>ProjectExec</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Sort-merge Join</p></td>
+<tr class="row-odd"><td><p>ShuffleExchangeExec</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Hash Join</p></td>
+<tr class="row-even"><td><p>ShuffledHashJoinExec</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>BroadcastHashJoinExec</p></td>
+<tr class="row-odd"><td><p>SortExec</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Shuffle</p></td>
+<tr class="row-even"><td><p>SortMergeJoinExec</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>Expand</p></td>
+<tr class="row-odd"><td><p>UnionExec</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>Union</p></td>
+<tr class="row-even"><td><p>WindowExec</p></td>
+<td><p>Yes</p></td>
<td><p></p></td>
</tr>
</tbody>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@datafusion.apache.org
For additional commands, e-mail: commits-h...@datafusion.apache.org
