This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new edd90044 Publish built docs triggered by
d584229c8e7d9f04b9df2803958b87714356860b
edd90044 is described below
commit edd90044077f84b3875cdf1a97dbe04e5d94f002
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Tue Jun 18 23:04:00 2024 +0000
Publish built docs triggered by d584229c8e7d9f04b9df2803958b87714356860b
---
_sources/contributor-guide/adding_a_new_expression.md.txt | 9 +++------
_sources/user-guide/installation.md.txt | 2 +-
_sources/user-guide/overview.md.txt | 2 +-
contributor-guide/adding_a_new_expression.html | 9 +++------
searchindex.js | 2 +-
user-guide/installation.html | 2 +-
user-guide/overview.html | 2 +-
7 files changed, 11 insertions(+), 17 deletions(-)
diff --git a/_sources/contributor-guide/adding_a_new_expression.md.txt
b/_sources/contributor-guide/adding_a_new_expression.md.txt
index 6cf10c75..6d906c66 100644
--- a/_sources/contributor-guide/adding_a_new_expression.md.txt
+++ b/_sources/contributor-guide/adding_a_new_expression.md.txt
@@ -46,7 +46,7 @@ The `QueryPlanSerde` object has a method `exprToProto`, which
is responsible for
For example, the `unhex` function looks like this:
```scala
-case e: Unhex if !isSpark32 =>
+case e: Unhex =>
val unHex = unhexSerde(e)
val childExpr = exprToProtoInternal(unHex._1, inputs)
@@ -59,7 +59,6 @@ case e: Unhex if !isSpark32 =>
A few things to note here:
-* The `isSpark32` check is used to fall back to Spark's implementation of
`unhex` in Spark 3.2. This is somewhat context specific, because in this case,
due to a bug in Spark 3.2 for `unhex`, we want to use the Spark implementation
and not a Comet implementation that would behave differently if correct.
* The function is recursively called on child expressions, so you'll need to
make sure that the child expressions are also converted to protobuf.
* `scalarExprToProtoWithReturnType` is for scalar functions that need return
type information. Your expression may use a different method depending on the
type of expression.
@@ -71,8 +70,6 @@ For example, this is the test case for the `unhex` expression:
```scala
test("unhex") {
- assume(!isSpark32, "unhex function has incorrect behavior in 3.2") // used
to skip the test in Spark 3.2
-
val table = "unhex_table"
withTable(table) {
sql(s"create table $table(col string) using parquet")
@@ -172,11 +169,11 @@ pub(super) fn spark_unhex(args: &[ColumnarValue]) ->
Result<ColumnarValue, DataF
If the expression you're adding has different behavior across different Spark
versions, you'll need to account for that in your implementation. There are two
tools at your disposal to help with this:
1. Shims that exist in
`spark/src/main/spark-$SPARK_VERSION/org/apache/comet/shims/CometExprShim.scala`
for each Spark version. These shims are used to provide compatibility between
different Spark versions.
-2. Variables that correspond to the Spark version, such as `isSpark32`, which
can be used to conditionally execute code based on the Spark version.
+2. Variables that correspond to the Spark version, such as `isSpark33Plus`,
which can be used to conditionally execute code based on the Spark version.
## Shimming to Support Different Spark Versions
-By adding shims for each Spark version, you can provide a consistent interface
for the expression across different Spark versions. For example, `unhex` added
a new optional parameter is Spark 3.4, for if it should `failOnError` or not.
So for version 3.2 and 3.3, the shim is:
+By adding shims for each Spark version, you can provide a consistent interface
for the expression across different Spark versions. For example, `unhex` added
a new optional parameter is Spark 3.4, for if it should `failOnError` or not.
So for version 3.3, the shim is:
```scala
trait CometExprShim {
diff --git a/_sources/user-guide/installation.md.txt
b/_sources/user-guide/installation.md.txt
index 7335a488..bdf6c0e0 100644
--- a/_sources/user-guide/installation.md.txt
+++ b/_sources/user-guide/installation.md.txt
@@ -28,7 +28,7 @@ Make sure the following requirements are met and software
installed on your mach
## Requirements
-- Apache Spark 3.2, 3.3, or 3.4
+- Apache Spark 3.3, or 3.4
- JDK 8 and up
- GLIBC 2.17 (Centos 7) and up
diff --git a/_sources/user-guide/overview.md.txt
b/_sources/user-guide/overview.md.txt
index b5425d77..87f5f286 100644
--- a/_sources/user-guide/overview.md.txt
+++ b/_sources/user-guide/overview.md.txt
@@ -40,7 +40,7 @@ The following diagram illustrates the architecture of Comet:
## Current Status
-The project is currently integrated into Apache Spark 3.2, 3.3, and 3.4.
+The project is currently integrated into Apache Spark 3.3, and 3.4.
## Feature Parity with Apache Spark
diff --git a/contributor-guide/adding_a_new_expression.html
b/contributor-guide/adding_a_new_expression.html
index 1c35910f..fcd012a6 100644
--- a/contributor-guide/adding_a_new_expression.html
+++ b/contributor-guide/adding_a_new_expression.html
@@ -394,7 +394,7 @@ under the License.
<h3>Adding the Expression in Scala<a class="headerlink"
href="#adding-the-expression-in-scala" title="Link to this heading">¶</a></h3>
<p>The <code class="docutils literal notranslate"><span
class="pre">QueryPlanSerde</span></code> object has a method <code
class="docutils literal notranslate"><span
class="pre">exprToProto</span></code>, which is responsible for converting a
Spark expression to a protobuf expression. Within that method is an <code
class="docutils literal notranslate"><span
class="pre">exprToProtoInternal</span></code> method that contains a large
match statement for each expression type. You’ll need to [...]
<p>For example, the <code class="docutils literal notranslate"><span
class="pre">unhex</span></code> function looks like this:</p>
-<div class="highlight-scala notranslate"><div
class="highlight"><pre><span></span><span class="k">case</span><span class="w">
</span><span class="n">e</span><span class="p">:</span><span class="w">
</span><span class="nc">Unhex</span><span class="w"> </span><span
class="k">if</span><span class="w"> </span><span class="o">!</span><span
class="n">isSpark32</span><span class="w"> </span><span class="o">=></span>
+<div class="highlight-scala notranslate"><div
class="highlight"><pre><span></span><span class="k">case</span><span class="w">
</span><span class="n">e</span><span class="p">:</span><span class="w">
</span><span class="nc">Unhex</span><span class="w"> </span><span
class="o">=></span>
<span class="w"> </span><span class="kd">val</span><span class="w">
</span><span class="n">unHex</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="n">unhexSerde</span><span class="p">(</span><span
class="n">e</span><span class="p">)</span>
<span class="w"> </span><span class="kd">val</span><span class="w">
</span><span class="n">childExpr</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="n">exprToProtoInternal</span><span class="p">(</span><span
class="n">unHex</span><span class="p">.</span><span class="n">_1</span><span
class="p">,</span><span class="w"> </span><span class="n">inputs</span><span
class="p">)</span>
@@ -407,7 +407,6 @@ under the License.
</div>
<p>A few things to note here:</p>
<ul class="simple">
-<li><p>The <code class="docutils literal notranslate"><span
class="pre">isSpark32</span></code> check is used to fall back to Spark’s
implementation of <code class="docutils literal notranslate"><span
class="pre">unhex</span></code> in Spark 3.2. This is somewhat context
specific, because in this case, due to a bug in Spark 3.2 for <code
class="docutils literal notranslate"><span class="pre">unhex</span></code>, we
want to use the Spark implementation and not a Comet implementation that [...]
<li><p>The function is recursively called on child expressions, so you’ll need
to make sure that the child expressions are also converted to protobuf.</p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">scalarExprToProtoWithReturnType</span></code> is for scalar
functions that need return type information. Your expression may use a
different method depending on the type of expression.</p></li>
</ul>
@@ -416,8 +415,6 @@ under the License.
<p>It is important to verify that the new expression is correctly recognized
by the native execution engine and matches the expected spark behavior. To do
this, you can add a set of test cases in the <code class="docutils literal
notranslate"><span class="pre">CometExpressionSuite</span></code>, and use the
<code class="docutils literal notranslate"><span
class="pre">checkSparkAnswerAndOperator</span></code> method to compare the
results of the new expression with the expected Spark resu [...]
<p>For example, this is the test case for the <code class="docutils literal
notranslate"><span class="pre">unhex</span></code> expression:</p>
<div class="highlight-scala notranslate"><div
class="highlight"><pre><span></span><span class="n">test</span><span
class="p">(</span><span class="s">"unhex"</span><span
class="p">)</span><span class="w"> </span><span class="p">{</span>
-<span class="w"> </span><span class="n">assume</span><span
class="p">(</span><span class="o">!</span><span class="n">isSpark32</span><span
class="p">,</span><span class="w"> </span><span class="s">"unhex function
has incorrect behavior in 3.2"</span><span class="p">)</span><span
class="w"> </span><span class="c1">// used to skip the test in Spark 3.2</span>
-
<span class="w"> </span><span class="kd">val</span><span class="w">
</span><span class="n">table</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="s">"unhex_table"</span>
<span class="w"> </span><span class="n">withTable</span><span
class="p">(</span><span class="n">table</span><span class="p">)</span><span
class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">sql</span><span
class="p">(</span><span class="s">s"create table </span><span
class="si">$</span><span class="n">table</span><span class="s">(col string)
using parquet"</span><span class="p">)</span>
@@ -508,13 +505,13 @@ under the License.
<p>If the expression you’re adding has different behavior across different
Spark versions, you’ll need to account for that in your implementation. There
are two tools at your disposal to help with this:</p>
<ol class="arabic simple">
<li><p>Shims that exist in <code class="docutils literal notranslate"><span
class="pre">spark/src/main/spark-$SPARK_VERSION/org/apache/comet/shims/CometExprShim.scala</span></code>
for each Spark version. These shims are used to provide compatibility between
different Spark versions.</p></li>
-<li><p>Variables that correspond to the Spark version, such as <code
class="docutils literal notranslate"><span class="pre">isSpark32</span></code>,
which can be used to conditionally execute code based on the Spark
version.</p></li>
+<li><p>Variables that correspond to the Spark version, such as <code
class="docutils literal notranslate"><span
class="pre">isSpark33Plus</span></code>, which can be used to conditionally
execute code based on the Spark version.</p></li>
</ol>
</section>
</section>
<section id="shimming-to-support-different-spark-versions">
<h2>Shimming to Support Different Spark Versions<a class="headerlink"
href="#shimming-to-support-different-spark-versions" title="Link to this
heading">¶</a></h2>
-<p>By adding shims for each Spark version, you can provide a consistent
interface for the expression across different Spark versions. For example,
<code class="docutils literal notranslate"><span
class="pre">unhex</span></code> added a new optional parameter is Spark 3.4,
for if it should <code class="docutils literal notranslate"><span
class="pre">failOnError</span></code> or not. So for version 3.2 and 3.3, the
shim is:</p>
+<p>By adding shims for each Spark version, you can provide a consistent
interface for the expression across different Spark versions. For example,
<code class="docutils literal notranslate"><span
class="pre">unhex</span></code> added a new optional parameter is Spark 3.4,
for if it should <code class="docutils literal notranslate"><span
class="pre">failOnError</span></code> or not. So for version 3.3, the shim
is:</p>
<div class="highlight-scala notranslate"><div
class="highlight"><pre><span></span><span class="k">trait</span><span
class="w"> </span><span class="nc">CometExprShim</span><span class="w">
</span><span class="p">{</span>
<span class="w"> </span><span class="cm">/**</span>
<span class="cm"> * Returns a tuple of expressions for the `unhex`
function.</span>
diff --git a/searchindex.js b/searchindex.js
index 7647b48f..57adf649 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"ANSI mode": [[8, "ansi-mode"]], "API
Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[7, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, "adding-a-new-expression"]], "Adding a New Scalar Function Expression":
[[0, "adding-a-new-scalar-function-expression"]], "Adding the Expression To the
Protobuf Definition" [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"ANSI mode": [[8, "ansi-mode"]], "API
Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[7, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, "adding-a-new-expression"]], "Adding a New Scalar Function Expression":
[[0, "adding-a-new-scalar-function-expression"]], "Adding the Expression To the
Protobuf Definition" [...]
\ No newline at end of file
diff --git a/user-guide/installation.html b/user-guide/installation.html
index 5925c0d7..3d0b01a6 100644
--- a/user-guide/installation.html
+++ b/user-guide/installation.html
@@ -357,7 +357,7 @@ under the License.
<section id="requirements">
<h2>Requirements<a class="headerlink" href="#requirements" title="Link to this
heading">¶</a></h2>
<ul class="simple">
-<li><p>Apache Spark 3.2, 3.3, or 3.4</p></li>
+<li><p>Apache Spark 3.3, or 3.4</p></li>
<li><p>JDK 8 and up</p></li>
<li><p>GLIBC 2.17 (Centos 7) and up</p></li>
</ul>
diff --git a/user-guide/overview.html b/user-guide/overview.html
index f74dd4f9..1adecd9a 100644
--- a/user-guide/overview.html
+++ b/user-guide/overview.html
@@ -342,7 +342,7 @@ Filter/Project/Aggregation/Join/Exchange etc.</p></li>
</section>
<section id="current-status">
<h2>Current Status<a class="headerlink" href="#current-status" title="Link to
this heading">¶</a></h2>
-<p>The project is currently integrated into Apache Spark 3.2, 3.3, and 3.4.</p>
+<p>The project is currently integrated into Apache Spark 3.3, and 3.4.</p>
</section>
<section id="feature-parity-with-apache-spark">
<h2>Feature Parity with Apache Spark<a class="headerlink"
href="#feature-parity-with-apache-spark" title="Link to this heading">¶</a></h2>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
