This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 39cbfd1d Publish built docs triggered by
e33d56050187c75afcbbdc2def96a093a019db6a
39cbfd1d is described below
commit 39cbfd1d5d28df8a84d438f3abe7466d193d656c
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Aug 2 12:40:29 2024 +0000
Publish built docs triggered by e33d56050187c75afcbbdc2def96a093a019db6a
---
_sources/user-guide/compatibility.md.txt | 6 ++++++
_sources/user-guide/configs.md.txt | 1 +
searchindex.js | 2 +-
user-guide/compatibility.html | 11 +++++++++++
user-guide/configs.html | 14 +++++++++-----
5 files changed, 28 insertions(+), 6 deletions(-)
diff --git a/_sources/user-guide/compatibility.md.txt
b/_sources/user-guide/compatibility.md.txt
index a16fd1b2..0af44eb6 100644
--- a/_sources/user-guide/compatibility.md.txt
+++ b/_sources/user-guide/compatibility.md.txt
@@ -32,6 +32,12 @@ be used in production.
There is an [epic](https://github.com/apache/datafusion-comet/issues/313)
where we are tracking the work to fully implement ANSI support.
+## Regular Expressions
+
+Comet uses the Rust regexp crate for evaluating regular expressions, and this
has different behavior from Java's
+regular expression engine. Comet will fall back to Spark for patterns that are
known to produce different results, but
+this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`.
+
## Cast
Cast operations in Comet fall into three levels of support:
diff --git a/_sources/user-guide/configs.md.txt
b/_sources/user-guide/configs.md.txt
index 0f648324..40304be5 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -44,6 +44,7 @@ Comet provides the following configuration settings.
| spark.comet.memory.overhead.min | Minimum amount of additional memory to be
allocated per executor process for Comet, in MiB. | 402653184b |
| spark.comet.nativeLoadRequired | Whether to require Comet native library to
load successfully when Comet is enabled. If not, Comet will silently fallback
to Spark when it fails to load the native lib. Otherwise, an error will be
thrown and the Spark job will be aborted. | false |
| spark.comet.parquet.enable.directBuffer | Whether to use Java direct byte
buffer when reading Parquet. By default, this is false | false |
+| spark.comet.regexp.allowIncompatible | Comet is not currently fully
compatible with Spark for all regular expressions. Set this config to true to
allow them anyway using Rust's regular expression engine. See compatibility
guide for more information. | false |
| spark.comet.rowToColumnar.supportedOperatorList | A comma-separated list of
row-based operators that will be converted to columnar format when
'spark.comet.rowToColumnar.enabled' is true | Range,InMemoryTableScan |
| spark.comet.scan.enabled | Whether to enable Comet scan. When this is turned
on, Spark will use Comet to read Parquet data source. Note that to enable
native vectorized execution, both this config and 'spark.comet.exec.enabled'
need to be enabled. By default, this config is true. | true |
| spark.comet.scan.preFetch.enabled | Whether to enable pre-fetching feature
of CometScan. By default is disabled. | false |
diff --git a/searchindex.js b/searchindex.js
index be094d48..f755c292 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[7, "install-comet"]], "2.
Clone Spark and Apply Diff": [[7, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[7, "run-spark-sql-tests"]], "ANSI mode": [[9, "ansi-mode"]], "API
Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[8, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, nu [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[7, "install-comet"]], "2.
Clone Spark and Apply Diff": [[7, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[7, "run-spark-sql-tests"]], "ANSI mode": [[9, "ansi-mode"]], "API
Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[8, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, nu [...]
\ No newline at end of file
diff --git a/user-guide/compatibility.html b/user-guide/compatibility.html
index 2c70a27a..1ef35481 100644
--- a/user-guide/compatibility.html
+++ b/user-guide/compatibility.html
@@ -271,6 +271,11 @@ under the License.
ANSI mode
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#regular-expressions">
+ Regular Expressions
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#cast">
Cast
@@ -351,6 +356,12 @@ specify <code class="docutils literal notranslate"><span
class="pre">spark.comet
be used in production.</p>
<p>There is an <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/313";>epic</a> where we
are tracking the work to fully implement ANSI support.</p>
</section>
+<section id="regular-expressions">
+<h2>Regular Expressions<a class="headerlink" href="#regular-expressions"
title="Link to this heading">¶</a></h2>
+<p>Comet uses the Rust regexp crate for evaluating regular expressions, and
this has different behavior from Java’s
+regular expression engine. Comet will fall back to Spark for patterns that are
known to produce different results, but
+this can be overridden by setting <code class="docutils literal
notranslate"><span
class="pre">spark.comet.regexp.allowIncompatible=true</span></code>.</p>
+</section>
<section id="cast">
<h2>Cast<a class="headerlink" href="#cast" title="Link to this
heading">¶</a></h2>
<p>Cast operations in Comet fall into three levels of support:</p>
diff --git a/user-guide/configs.html b/user-guide/configs.html
index 092c46a7..719e423a 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -402,23 +402,27 @@ under the License.
<td><p>Whether to use Java direct byte buffer when reading Parquet. By
default, this is false</p></td>
<td><p>false</p></td>
</tr>
-<tr
class="row-odd"><td><p>spark.comet.rowToColumnar.supportedOperatorList</p></td>
+<tr class="row-odd"><td><p>spark.comet.regexp.allowIncompatible</p></td>
+<td><p>Comet is not currently fully compatible with Spark for all regular
expressions. Set this config to true to allow them anyway using Rust’s regular
expression engine. See compatibility guide for more information.</p></td>
+<td><p>false</p></td>
+</tr>
+<tr
class="row-even"><td><p>spark.comet.rowToColumnar.supportedOperatorList</p></td>
<td><p>A comma-separated list of row-based operators that will be converted to
columnar format when ‘spark.comet.rowToColumnar.enabled’ is true</p></td>
<td><p>Range,InMemoryTableScan</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.scan.enabled</p></td>
+<tr class="row-odd"><td><p>spark.comet.scan.enabled</p></td>
<td><p>Whether to enable Comet scan. When this is turned on, Spark will use
Comet to read Parquet data source. Note that to enable native vectorized
execution, both this config and ‘spark.comet.exec.enabled’ need to be enabled.
By default, this config is true.</p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.scan.preFetch.enabled</p></td>
+<tr class="row-even"><td><p>spark.comet.scan.preFetch.enabled</p></td>
<td><p>Whether to enable pre-fetching feature of CometScan. By default is
disabled.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.scan.preFetch.threadNum</p></td>
+<tr class="row-odd"><td><p>spark.comet.scan.preFetch.threadNum</p></td>
<td><p>The number of threads running pre-fetching for CometScan. Effective if
spark.comet.scan.preFetch.enabled is enabled. By default it is 2. Note that
more pre-fetching threads means more memory requirement to store pre-fetched
row groups.</p></td>
<td><p>2</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.shuffle.preferDictionary.ratio</p></td>
+<tr class="row-even"><td><p>spark.comet.shuffle.preferDictionary.ratio</p></td>
<td><p>The ratio of total values to distinct values in a string column to
decide whether to prefer dictionary encoding when shuffling the column. If the
ratio is higher than this config, dictionary encoding will be used on shuffling
string column. This config is effective if it is higher than 1.0. By default,
this config is 10.0. Note that this config is only used when <code
class="docutils literal notranslate"><span
class="pre">spark.comet.exec.shuffle.mode</span></code> is <code class= [...]
<td><p>10.0</p></td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
