This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 11ea5a41d Publish built docs triggered by
6d5bdbe2d08ed0e2488a1f8406be15f057ae68d3
11ea5a41d is described below
commit 11ea5a41dd1bf144c89138ac130b3fc2f7c914bb
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon Oct 6 19:57:40 2025 +0000
Publish built docs triggered by 6d5bdbe2d08ed0e2488a1f8406be15f057ae68d3
---
_sources/user-guide/0.10/configs.md.txt | 2 +-
_sources/user-guide/0.10/tuning.md.txt | 6 +++---
_sources/user-guide/latest/expressions.md.txt | 7 ++++---
searchindex.js | 2 +-
user-guide/0.10/configs.html | 2 +-
user-guide/0.10/tuning.html | 6 +++---
user-guide/latest/expressions.html | 30 +++++++++++++++------------
7 files changed, 30 insertions(+), 25 deletions(-)
diff --git a/_sources/user-guide/0.10/configs.md.txt
b/_sources/user-guide/0.10/configs.md.txt
index c923c5668..6f3e32595 100644
--- a/_sources/user-guide/0.10/configs.md.txt
+++ b/_sources/user-guide/0.10/configs.md.txt
@@ -49,7 +49,7 @@ Comet provides the following configuration settings.
| spark.comet.exec.globalLimit.enabled | Whether to enable globalLimit by
default. | true |
| spark.comet.exec.hashJoin.enabled | Whether to enable hashJoin by default. |
true |
| spark.comet.exec.localLimit.enabled | Whether to enable localLimit by
default. | true |
-| spark.comet.exec.memoryPool | The type of memory pool to be used for Comet
native execution. When running Spark in on-heap mode, available pool types are
'greedy', 'fair_spill', 'greedy_task_shared', 'fair_spill_task_shared',
'greedy_global', 'fair_spill_global', and `unbounded`. When running Spark in
off-heap mode, available pool types are 'unified' and `fair_unified`. The
default pool type is `greedy_task_shared` for on-heap mode and `unified` for
off-heap mode. For more information, [...]
+| spark.comet.exec.memoryPool | The type of memory pool to be used for Comet
native execution. When running Spark in on-heap mode, available pool types are
'greedy', 'fair_spill', 'greedy_task_shared', 'fair_spill_task_shared',
'greedy_global', 'fair_spill_global', and `unbounded`. When running Spark in
off-heap mode, available pool types are 'greedy_unified' and `fair_unified`.
The default pool type is `greedy_task_shared` for on-heap mode and `unified`
for off-heap mode. For more infor [...]
| spark.comet.exec.project.enabled | Whether to enable project by default. |
true |
| spark.comet.exec.replaceSortMergeJoin | Experimental feature to force Spark
to replace SortMergeJoin with ShuffledHashJoin for improved performance. This
feature is not stable yet. For more information, refer to the Comet Tuning
Guide (https://datafusion.apache.org/comet/user-guide/tuning.html). | false |
| spark.comet.exec.shuffle.compression.codec | The codec of Comet native
shuffle used to compress shuffle data. lz4, zstd, and snappy are supported.
Compression can be disabled by setting spark.shuffle.compress=false. | lz4 |
diff --git a/_sources/user-guide/0.10/tuning.md.txt
b/_sources/user-guide/0.10/tuning.md.txt
index 04f0c5523..b56cb8171 100644
--- a/_sources/user-guide/0.10/tuning.md.txt
+++ b/_sources/user-guide/0.10/tuning.md.txt
@@ -119,13 +119,13 @@ Comet implements multiple memory pool implementations.
The type of pool can be s
The valid pool types for off-heap mode are:
-- `unified` (default when `spark.memory.offHeap.enabled=true` is set)
-- `fair_unified`
+- `fair_unified` (default when `spark.memory.offHeap.enabled=true` is set)
+- `greedy_unified`
Both of these pools share off-heap memory between Spark and Comet. This
approach is referred to as
unified memory management. The size of the pool is specified by
`spark.memory.offHeap.size`.
-The `unified` pool type implements a greedy first-come first-serve limit. This
pool works well for queries that do not
+The `greedy_unified` pool type implements a greedy first-come first-serve
limit. This pool works well for queries that do not
need to spill or have a single spillable operator.
The `fair_unified` pool type prevents operators from using more than an even
fraction of the available memory
diff --git a/_sources/user-guide/latest/expressions.md.txt
b/_sources/user-guide/latest/expressions.md.txt
index 4dbcec6f0..e727642dc 100644
--- a/_sources/user-guide/latest/expressions.md.txt
+++ b/_sources/user-guide/latest/expressions.md.txt
@@ -61,7 +61,7 @@ incompatible expressions.
## String Functions
| Expression | Spark-Compatible? | Compatibility Notes
|
-|-----------------| ----------------- |
----------------------------------------------------------------------------------------------------------
|
+| --------------- | ----------------- |
----------------------------------------------------------------------------------------------------------
|
| Ascii | Yes |
|
| BitLength | Yes |
|
| Chr | Yes |
|
@@ -93,7 +93,7 @@ incompatible expressions.
## Date/Time Functions
| Expression | SQL | Spark-Compatible? |
Compatibility Notes
|
-|----------------|------------------------------| -----------------
|----------------------------------------------------------------------------------------------------------------------|
+| -------------- | ---------------------------- | ----------------- |
--------------------------------------------------------------------------------------------------------------------
|
| DateAdd | `date_add` | Yes |
|
| DateSub | `date_sub` | Yes |
|
| DatePart | `date_part(field, source)` | Yes |
Supported values of `field`:
`year`/`month`/`week`/`day`/`dayofweek`/`dayofweek_iso`/`doy`/`quarter`/`hour`/`minute`
|
@@ -116,7 +116,7 @@ incompatible expressions.
## Math Expressions
| Expression | SQL | Spark-Compatible? | Compatibility Notes
|
-|----------------|-----------|-------------------|-----------------------------------|
+| -------------- | --------- | ----------------- |
--------------------------------- |
| Acos | `acos` | Yes |
|
| Add | `+` | Yes |
|
| Asin | `asin` | Yes |
|
@@ -208,6 +208,7 @@ incompatible expressions.
| ArrayContains | Yes |
|
| ArrayDistinct | No | Behaves differently than spark. Comet
first sorts then removes duplicates while Spark preserves the original order.
|
| ArrayExcept | No |
|
+| ArrayFilter | Yes | Only supports case where function is
`IsNotNull`
|
| ArrayInsert | No |
|
| ArrayIntersect | No |
|
| ArrayJoin | No |
|
diff --git a/searchindex.js b/searchindex.js
index 5a7b3e465..cab25bad4 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[12, "install-comet"]],
"2. Clone Spark and Apply Diff": [[12, "clone-spark-and-apply-diff"]], "3. Run
Spark SQL Tests": [[12, "run-spark-sql-tests"]], "ANSI Mode": [[17,
"ansi-mode"], [56, "ansi-mode"]], "ANSI mode": [[30, "ansi-mode"], [43,
"ansi-mode"]], "API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "Accelerating Apache Iceberg
Parquet Scans using Comet (Experimental)": [[22, null], [35, n [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[12, "install-comet"]],
"2. Clone Spark and Apply Diff": [[12, "clone-spark-and-apply-diff"]], "3. Run
Spark SQL Tests": [[12, "run-spark-sql-tests"]], "ANSI Mode": [[17,
"ansi-mode"], [56, "ansi-mode"]], "ANSI mode": [[30, "ansi-mode"], [43,
"ansi-mode"]], "API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "Accelerating Apache Iceberg
Parquet Scans using Comet (Experimental)": [[22, null], [35, n [...]
\ No newline at end of file
diff --git a/user-guide/0.10/configs.html b/user-guide/0.10/configs.html
index ee958f93a..d7c4d36af 100644
--- a/user-guide/0.10/configs.html
+++ b/user-guide/0.10/configs.html
@@ -688,7 +688,7 @@ under the License.
<td><p>true</p></td>
</tr>
<tr class="row-odd"><td><p>spark.comet.exec.memoryPool</p></td>
-<td><p>The type of memory pool to be used for Comet native execution. When
running Spark in on-heap mode, available pool types are ‘greedy’, ‘fair_spill’,
‘greedy_task_shared’, ‘fair_spill_task_shared’, ‘greedy_global’,
‘fair_spill_global’, and <code class="docutils literal notranslate"><span
class="pre">unbounded</span></code>. When running Spark in off-heap mode,
available pool types are ‘unified’ and <code class="docutils literal
notranslate"><span class="pre">fair_unified</span></cod [...]
+<td><p>The type of memory pool to be used for Comet native execution. When
running Spark in on-heap mode, available pool types are ‘greedy’, ‘fair_spill’,
‘greedy_task_shared’, ‘fair_spill_task_shared’, ‘greedy_global’,
‘fair_spill_global’, and <code class="docutils literal notranslate"><span
class="pre">unbounded</span></code>. When running Spark in off-heap mode,
available pool types are ‘greedy_unified’ and <code class="docutils literal
notranslate"><span class="pre">fair_unified</spa [...]
<td><p>default</p></td>
</tr>
<tr class="row-even"><td><p>spark.comet.exec.project.enabled</p></td>
diff --git a/user-guide/0.10/tuning.html b/user-guide/0.10/tuning.html
index 98489c4b9..30c29343e 100644
--- a/user-guide/0.10/tuning.html
+++ b/user-guide/0.10/tuning.html
@@ -777,12 +777,12 @@ resource managers respect Apache Spark memory
configuration before starting the
<p>Comet implements multiple memory pool implementations. The type of pool can
be specified with <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.memoryPool</span></code>.</p>
<p>The valid pool types for off-heap mode are:</p>
<ul class="simple">
-<li><p><code class="docutils literal notranslate"><span
class="pre">unified</span></code> (default when <code class="docutils literal
notranslate"><span class="pre">spark.memory.offHeap.enabled=true</span></code>
is set)</p></li>
-<li><p><code class="docutils literal notranslate"><span
class="pre">fair_unified</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">fair_unified</span></code> (default when <code class="docutils
literal notranslate"><span
class="pre">spark.memory.offHeap.enabled=true</span></code> is set)</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">greedy_unified</span></code></p></li>
</ul>
<p>Both of these pools share off-heap memory between Spark and Comet. This
approach is referred to as
unified memory management. The size of the pool is specified by <code
class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.size</span></code>.</p>
-<p>The <code class="docutils literal notranslate"><span
class="pre">unified</span></code> pool type implements a greedy first-come
first-serve limit. This pool works well for queries that do not
+<p>The <code class="docutils literal notranslate"><span
class="pre">greedy_unified</span></code> pool type implements a greedy
first-come first-serve limit. This pool works well for queries that do not
need to spill or have a single spillable operator.</p>
<p>The <code class="docutils literal notranslate"><span
class="pre">fair_unified</span></code> pool type prevents operators from using
more than an even fraction of the available memory
(i.e. <code class="docutils literal notranslate"><span
class="pre">pool_size</span> <span class="pre">/</span> <span
class="pre">num_reservations</span></code>). This pool works best when you know
beforehand
diff --git a/user-guide/latest/expressions.html
b/user-guide/latest/expressions.html
index 2697fc42e..fbd631cf7 100644
--- a/user-guide/latest/expressions.html
+++ b/user-guide/latest/expressions.html
@@ -1372,55 +1372,59 @@ incompatible expressions.</p>
<td><p>No</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>ArrayInsert</p></td>
+<tr class="row-odd"><td><p>ArrayFilter</p></td>
+<td><p>Yes</p></td>
+<td><p>Only supports case where function is <code class="docutils literal
notranslate"><span class="pre">IsNotNull</span></code></p></td>
+</tr>
+<tr class="row-even"><td><p>ArrayInsert</p></td>
<td><p>No</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>ArrayIntersect</p></td>
+<tr class="row-odd"><td><p>ArrayIntersect</p></td>
<td><p>No</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>ArrayJoin</p></td>
+<tr class="row-even"><td><p>ArrayJoin</p></td>
<td><p>No</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>ArrayMax</p></td>
+<tr class="row-odd"><td><p>ArrayMax</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>ArrayMin</p></td>
+<tr class="row-even"><td><p>ArrayMin</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>ArrayRemove</p></td>
+<tr class="row-odd"><td><p>ArrayRemove</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>ArrayRepeat</p></td>
+<tr class="row-even"><td><p>ArrayRepeat</p></td>
<td><p>No</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>ArrayUnion</p></td>
+<tr class="row-odd"><td><p>ArrayUnion</p></td>
<td><p>No</p></td>
<td><p>Behaves differently than spark. Comet sorts the input arrays before
performing the union, while Spark preserves the order of the first array and
appends unique elements from the second.</p></td>
</tr>
-<tr class="row-odd"><td><p>ArraysOverlap</p></td>
+<tr class="row-even"><td><p>ArraysOverlap</p></td>
<td><p>No</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p>CreateArray</p></td>
+<tr class="row-odd"><td><p>CreateArray</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>ElementAt</p></td>
+<tr class="row-even"><td><p>ElementAt</p></td>
<td><p>Yes</p></td>
<td><p>Input must be an array. Map inputs are not supported.</p></td>
</tr>
-<tr class="row-even"><td><p>Flatten</p></td>
+<tr class="row-odd"><td><p>Flatten</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p>GetArrayItem</p></td>
+<tr class="row-even"><td><p>GetArrayItem</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
