This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6b539dc6 Publish built docs triggered by
7a6f47f8a3d16c2b94009e633f38ea40d7f573ee
6b539dc6 is described below
commit 6b539dc6abb1994769035032d9d07274a083720d
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Sun Sep 22 21:10:16 2024 +0000
Publish built docs triggered by 7a6f47f8a3d16c2b94009e633f38ea40d7f573ee
---
_sources/user-guide/compatibility.md.txt | 8 ++++++++
searchindex.js | 2 +-
user-guide/compatibility.html | 13 +++++++++++++
3 files changed, 22 insertions(+), 1 deletion(-)
diff --git a/_sources/user-guide/compatibility.md.txt
b/_sources/user-guide/compatibility.md.txt
index 5e849992..031322e9 100644
--- a/_sources/user-guide/compatibility.md.txt
+++ b/_sources/user-guide/compatibility.md.txt
@@ -38,6 +38,14 @@ Comet uses the Rust regexp crate for evaluating regular
expressions, and this ha
regular expression engine. Comet will fall back to Spark for patterns that are
known to produce different results, but
this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`.
+## Floating number comparison
+
+Spark normalizes NaN and zero for floating point numbers for several cases.
See `NormalizeFloatingNumbers` optimization rule in Spark.
+However, one exception is comparison. Spark does not normalize NaN and zero
when comparing values
+because they are handled well in Spark (e.g.,
`SQLOrderingUtil.compareFloats`). But the comparison
+functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g.,
[arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
+So Comet will add additional normalization expression of NaN and zero for
comparison.
+
## Cast
Cast operations in Comet fall into three levels of support:
diff --git a/searchindex.js b/searchindex.js
index b023388c..80b0bdf0 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2.
Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]],
"API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2.
Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]],
"API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, [...]
\ No newline at end of file
diff --git a/user-guide/compatibility.html b/user-guide/compatibility.html
index 103fd643..4a0578fc 100644
--- a/user-guide/compatibility.html
+++ b/user-guide/compatibility.html
@@ -281,6 +281,11 @@ under the License.
Regular Expressions
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#floating-number-comparison">
+ Floating number comparison
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#cast">
Cast
@@ -367,6 +372,14 @@ be used in production.</p>
regular expression engine. Comet will fall back to Spark for patterns that are
known to produce different results, but
this can be overridden by setting <code class="docutils literal
notranslate"><span
class="pre">spark.comet.regexp.allowIncompatible=true</span></code>.</p>
</section>
+<section id="floating-number-comparison">
+<h2>Floating number comparison<a class="headerlink"
href="#floating-number-comparison" title="Link to this heading">¶</a></h2>
+<p>Spark normalizes NaN and zero for floating point numbers for several cases.
See <code class="docutils literal notranslate"><span
class="pre">NormalizeFloatingNumbers</span></code> optimization rule in Spark.
+However, one exception is comparison. Spark does not normalize NaN and zero
when comparing values
+because they are handled well in Spark (e.g., <code class="docutils literal
notranslate"><span class="pre">SQLOrderingUtil.compareFloats</span></code>).
But the comparison
+functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g.,
<a class="reference external"
href="https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#";>arrow::compute::kernels::cmp::eq</a>).
+So Comet will add additional normalization expression of NaN and zero for
comparison.</p>
+</section>
<section id="cast">
<h2>Cast<a class="headerlink" href="#cast" title="Link to this
heading">¶</a></h2>
<p>Cast operations in Comet fall into three levels of support:</p>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
