This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 511a675 Publish built docs triggered by
a340748c3ba526e6994b8e00248e261b8be9abe5
511a675 is described below
commit 511a675886ca178e975feb7ec1468ba05ec48999
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Apr 26 22:43:45 2024 +0000
Publish built docs triggered by a340748c3ba526e6994b8e00248e261b8be9abe5
---
_sources/contributor-guide/contributing.md.txt | 52 +++
_sources/contributor-guide/debugging.md.txt | 164 +++++++
_sources/contributor-guide/development.md.txt | 96 +++++
_sources/index.rst.txt | 15 +-
_sources/{ => user-guide}/compatibility.md.txt | 2 +-
_sources/user-guide/expressions.md.txt | 109 +++++
.../contributing.html | 148 +++++--
contributor-guide/debugging.html | 476 +++++++++++++++++++++
contributor-guide/development.html | 447 +++++++++++++++++++
genindex.html | 31 +-
index.html | 53 ++-
objects.inv | Bin 317 -> 405 bytes
search.html | 31 +-
searchindex.js | 2 +-
.../compatibility.html | 83 ++--
compatibility.html => user-guide/expressions.html | 233 +++++++---
16 files changed, 1795 insertions(+), 147 deletions(-)
diff --git a/_sources/contributor-guide/contributing.md.txt
b/_sources/contributor-guide/contributing.md.txt
new file mode 100644
index 0000000..2262692
--- /dev/null
+++ b/_sources/contributor-guide/contributing.md.txt
@@ -0,0 +1,52 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Contributing to Apache DataFusion Comet
+
+We welcome contributions to Comet in many areas, and encourage new
contributors to get involved.
+
+Here are some areas where you can help:
+
+- Testing Comet with existing Spark jobs and reporting issues for any bugs or
performance issues
+- Contributing code to support Spark expressions, operators, and data types
that are not currently supported
+- Reviewing pull requests and helping to test new features for correctness and
performance
+- Improving documentation
+
+## Finding issues to work on
+
+We maintain a list of good first issues in GitHub
[here](https://github.com/apache/datafusion-comet/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22).
+
+## Reporting issues
+
+We use [GitHub issues](https://github.com/apache/datafusion-comet/issues) for
bug reports and feature requests.
+
+## Asking for Help
+
+The Comet project uses the same Slack and Discord channels as the main Apache
DataFusion project. See details at
+[Apache DataFusion Communications]. There are dedicated Comet channels in both
Slack and Discord.
+
+## Regular public meetings
+
+The Comet contributors hold regular video calls where new and current
contributors are welcome to ask questions and
+coordinate on issues that they are working on.
+
+See the [Apache DataFusion Comet community meeting] Google document for more
information.
+
+[Apache DataFusion Communications]:
https://datafusion.apache.org/contributor-guide/communication.html
+[Apache DataFusion Comet community meeting]:
https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing
diff --git a/_sources/contributor-guide/debugging.md.txt
b/_sources/contributor-guide/debugging.md.txt
new file mode 100644
index 0000000..3b20ed0
--- /dev/null
+++ b/_sources/contributor-guide/debugging.md.txt
@@ -0,0 +1,164 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Comet Debugging Guide
+
+This HOWTO describes how to debug JVM code and Native code concurrently. The
guide assumes you have:
+
+1. Intellij as the Java IDE
+2. CLion as the Native IDE. For Rust code, the CLion Rust language plugin is
required. Note that the
+ Intellij Rust plugin is not sufficient.
+3. CLion/LLDB as the native debugger. CLion ships with a bundled LLDB and the
Rust community has
+ its own packaging of LLDB (`lldb-rust`). Both provide a better display of
Rust symbols than plain
+ LLDB or the LLDB that is bundled with XCode. We will use the LLDB packaged
with CLion for this guide.
+4. We will use a Comet _unit_ test as the canonical use case.
+
+_Caveat: The steps here have only been tested with JDK 11_ on Mac (M1)
+
+## Debugging for Advanced Developers
+
+Add a `.lldbinit` to comet/core. This is not strictly necessary but will be
useful if you want to
+use advanced `lldb` debugging.
+
+### In Intellij
+
+1. Set a breakpoint in `NativeBase.load()`, at a point _after_ the Comet
library has been loaded.
+
+1. Add a Debug Configuration for the unit test
+
+1. In the Debug Configuration for that unit test add `-Xint` as a JVM
parameter. This option is
+ undocumented _magic_. Without this, the LLDB debugger hits a EXC_BAD_ACCESS
(or EXC_BAD_INSTRUCTION) from
+ which one cannot recover.
+
+1. Add a println to the unit test to print the PID of the JVM process. (jps
can also be used but this is less error prone if you have multiple jvm
processes running)
+
+ ```JDK8
+ println("Waiting for Debugger: PID - ",
ManagementFactory.getRuntimeMXBean().getName())
+ ```
+
+ This will print something like : `PID@your_machine_name`.
+
+ For JDK9 and newer
+
+ ```JDK9
+ println("Waiting for Debugger: PID - ", ProcessHandle.current.pid)
+ ```
+
+ ==> Note the PID
+
+1. Debug-run the test in Intellij and wait for the breakpoint to be hit
+
+### In CLion
+
+1. After the breakpoint is hit in Intellij, in Clion (or LLDB from terminal or
editor) -
+
+ 1. Attach to the jvm process (make sure the PID matches). In CLion, this is
`Run -> Atttach to process`
+
+ 1. Put your breakpoint in the native code
+
+1. Go back to intellij and resume the process.
+
+1. Most debugging in CLion is similar to Intellij. For advanced LLDB based
debugging the LLDB command line can be accessed from the LLDB tab in the
Debugger view. Refer to the [LLDB
manual](https://lldb.llvm.org/use/tutorial.html) for LLDB commands.
+
+### After your debugging is done,
+
+1. In CLion, detach from the process if not already detached
+
+2. In Intellij, the debugger might have lost track of the process. If so, the
debugger tab
+ will show the process as running (even if the test/job is shown as
completed).
+
+3. Close the debugger tab, and if the IDS asks whether it should terminate the
process,
+ click Yes.
+
+4. In terminal, use jps to identify the process with the process id you were
debugging. If
+ it shows up as running, kill -9 [pid]. If that doesn't remove the process,
don't bother,
+ the process will be left behind as a zombie and will consume no
(significant) resources.
+ Eventually it will be cleaned up when you reboot possibly after a software
update.
+
+### Additional Info
+
+OpenJDK mailing list on debugging the JDK on MacOS
+https://mail.openjdk.org/pipermail/hotspot-dev/2019-September/039429.html
+
+Detecting the debugger
+https://stackoverflow.com/questions/5393403/can-a-java-application-detect-that-a-debugger-is-attached#:~:text=No.,to%20let%20your%20app%20continue.&text=I%20know%20that%20those%20are,meant%20with%20my%20first%20phrase).
+
+# Verbose debug
+
+By default, Comet outputs the exception details specific for Comet.
+
+```scala
+scala> spark.sql("my_failing_query").show(false)
+
+24/03/05 17:00:07 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)/
1]
+org.apache.comet.CometNativeException: Internal error: MIN/MAX is not expected
to receive scalars of incompatible types (Date32("NULL"), Int32(15901)).
+This was likely caused by a bug in DataFusion's code and we would welcome that
you file an bug report in our issue tracker
+ at org.apache.comet.Native.executePlan(Native Method)
+ at
org.apache.comet.CometExecIterator.executeNative(CometExecIterator.scala:65)
+ at
org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:111)
+ at
org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:126)
+
+```
+
+There is a verbose exception option by leveraging DataFusion
[backtraces](https://arrow.apache.org/datafusion/user-guide/example-usage.html#enable-backtraces)
+This option allows to append native DataFusion stacktrace to the original
error message.
+To enable this option with Comet it is needed to include `backtrace` feature
in
[Cargo.toml](https://github.com/apache/arrow-datafusion-comet/blob/main/core/Cargo.toml)
for DataFusion dependencies
+
+```
+datafusion-common = { version = "36.0.0", features = ["backtrace"] }
+datafusion = { default-features = false, version = "36.0.0", features =
["unicode_expressions", "backtrace"] }
+```
+
+Then build the Comet as
[described](https://github.com/apache/arrow-datafusion-comet/blob/main/README.md#getting-started)
+
+Start Comet with `RUST_BACKTRACE=1`
+
+```commandline
+RUST_BACKTRACE=1 $SPARK_HOME/spark-shell --jars
spark/target/comet-spark-spark3.4_2.12-0.1.0-SNAPSHOT.jar --conf
spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions --conf
spark.comet.enabled=true --conf spark.comet.exec.enabled=true --conf
spark.comet.exec.all.enabled=true
+```
+
+Get the expanded exception details
+
+```scala
+scala> spark.sql("my_failing_query").show(false)
+24/03/05 17:00:49 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
+org.apache.comet.CometNativeException: Internal error: MIN/MAX is not expected
to receive scalars of incompatible types (Date32("NULL"), Int32(15901))
+
+backtrace:
+ 0: std::backtrace::Backtrace::create
+ 1: datafusion_physical_expr::aggregate::min_max::min
+ 2: <datafusion_physical_expr::aggregate::min_max::MinAccumulator as
datafusion_expr::accumulator::Accumulator>::update_batch
+ 3: <futures_util::stream::stream::fuse::Fuse<S> as
futures_core::stream::Stream>::poll_next
+ 4:
comet::execution::jni_api::Java_org_apache_comet_Native_executePlan::{{closure}}
+ 5: _Java_org_apache_comet_Native_executePlan
+ (reduced)
+
+This was likely caused by a bug in DataFusion's code and we would welcome that
you file an bug report in our issue tracker
+ at org.apache.comet.Native.executePlan(Native Method)
+at org.apache.comet.CometExecIterator.executeNative(CometExecIterator.scala:65)
+at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:111)
+at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:126)
+(reduced)
+
+```
+
+Note:
+
+- The backtrace coverage in DataFusion is still improving. So there is a
chance the error still not covered, if so feel free to file a
[ticket](https://github.com/apache/arrow-datafusion/issues)
+- The backtrace evaluation comes with performance cost and intended mostly for
debugging purposes
diff --git a/_sources/contributor-guide/development.md.txt
b/_sources/contributor-guide/development.md.txt
new file mode 100644
index 0000000..63146c1
--- /dev/null
+++ b/_sources/contributor-guide/development.md.txt
@@ -0,0 +1,96 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Comet Development Guide
+
+## Project Layout
+
+```
+├── common <- common Java/Scala code
+├── conf <- configuration files
+├── core <- core native code, in Rust
+├── spark <- Spark integration
+```
+
+## Development Setup
+
+1. Make sure `JAVA_HOME` is set and point to JDK 8/11/17 installation.
+2. Install Rust toolchain. The easiest way is to use
+ [rustup](https://rustup.rs).
+
+## Build & Test
+
+A few common commands are specified in project's `Makefile`:
+
+- `make`: compile the entire project, but don't run tests
+- `make test-rust`: compile the project and run tests in Rust side
+- `make test-java`: compile the project and run tests in Java side
+- `make test`: compile the project and run tests in both Rust and Java
+ side.
+- `make release`: compile the project and creates a release build. This
+ is useful when you want to test Comet local installation in another project
+ such as Spark.
+- `make clean`: clean up the workspace
+- `bin/comet-spark-shell -d . -o spark/target/` run Comet spark shell for V1
datasources
+- `bin/comet-spark-shell -d . -o spark/target/ --conf
spark.sql.sources.useV1SourceList=""` run Comet spark shell for V2 datasources
+
+## Development Environment
+
+Comet is a multi-language project with native code written in Rust and JVM
code written in Java and Scala.
+For Rust code, the CLion IDE is recommended. For JVM code, IntelliJ IDEA is
recommended.
+
+Before opening the project in an IDE, make sure to run `make` first to
generate the necessary files for the IDEs. Currently, it's mostly about
+generating protobuf message classes for the JVM side. It's only required to
run `make` once after cloning the repo.
+
+### IntelliJ IDEA
+
+First make sure to install the Scala plugin in IntelliJ IDEA.
+After that, you can open the project in IntelliJ IDEA. The IDE should
automatically detect the project structure and import as a Maven project.
+
+### CLion
+
+First make sure to install the Rust plugin in CLion or you can use the
dedicated Rust IDE: RustRover.
+After that you can open the project in CLion. The IDE should automatically
detect the project structure and import as a Cargo project.
+
+### Running Tests in IDEA
+
+Like other Maven projects, you can run tests in IntelliJ IDEA by
right-clicking on the test class or test method and selecting "Run" or "Debug".
+However if the tests is related to the native side. Please make sure to run
`make core` or `cd core && cargo build` before running the tests in IDEA.
+
+## Benchmark
+
+There's a `make` command to run micro benchmarks in the repo. For
+instance:
+
+```
+make benchmark-org.apache.spark.sql.benchmark.CometReadBenchmark
+```
+
+To run TPC-H or TPC-DS micro benchmarks, please follow the instructions
+in the respective source code, e.g., `CometTPCHQueryBenchmark`.
+
+## Debugging
+
+Comet is a multi-language project with native code written in Rust and JVM
code written in Java and Scala.
+It is possible to debug both native and JVM code concurrently as described in
the [DEBUGGING guide](debugging)
+
+## Submitting a Pull Request
+
+Comet uses `cargo fmt`, [Scalafix](https://github.com/scalacenter/scalafix)
and [Spotless](https://github.com/diffplug/spotless/tree/main/plugin-maven) to
+automatically format the code. Before submitting a pull request, you can
simply run `make format` to format the code.
diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt
index c8f2735..4462a8d 100644
--- a/_sources/index.rst.txt
+++ b/_sources/index.rst.txt
@@ -35,16 +35,23 @@ Apache DataFusion Comet
Apache DataFusion Comet is an Apache Spark plugin that uses Apache DataFusion
as a native runtime to achieve improvement in terms of query efficiency and
query runtime.
-This documentation site is currently being developed. The most up-to-date
documentation can be found in the
-GitHub repository at https://github.com/apache/datafusion-comet.
+.. _toc.links:
+.. toctree::
+ :maxdepth: 1
+ :caption: User Guide
+
+ Supported Expressions <user-guide/expressions>
+ user-guide/compatibility
.. _toc.links:
.. toctree::
:maxdepth: 1
- :caption: Project Links
+ :caption: Contributor Guide
- compatibility
+ Getting Started <contributor-guide/contributing>
Github and Issue Tracker <https://github.com/apache/datafusion-comet>
+ contributor-guide/development
+ contributor-guide/debugging
.. _toc.asf-links:
.. toctree::
diff --git a/_sources/compatibility.md.txt
b/_sources/user-guide/compatibility.md.txt
similarity index 97%
rename from _sources/compatibility.md.txt
rename to _sources/user-guide/compatibility.md.txt
index 6e69f84..d817ba5 100644
--- a/_sources/compatibility.md.txt
+++ b/_sources/user-guide/compatibility.md.txt
@@ -34,7 +34,7 @@ There is an
[epic](https://github.com/apache/datafusion-comet/issues/313) where
## Cast
-Comet currently delegates to Apache DataFusion for most cast operations, and
this means that the behavior is not
+Comet currently delegates to Apache DataFusion for most cast operations, and
this means that the behavior is not
guaranteed to be consistent with Spark.
There is an [epic](https://github.com/apache/datafusion-comet/issues/286)
where we are tracking the work to implement Spark-compatible cast expressions.
diff --git a/_sources/user-guide/expressions.md.txt
b/_sources/user-guide/expressions.md.txt
new file mode 100644
index 0000000..f67a4ea
--- /dev/null
+++ b/_sources/user-guide/expressions.md.txt
@@ -0,0 +1,109 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Supported Spark Expressions
+
+The following Spark expressions are currently available:
+
+- Literals
+- Arithmetic Operators
+ - UnaryMinus
+ - Add/Minus/Multiply/Divide/Remainder
+- Conditional functions
+ - Case When
+ - If
+- Cast
+- Coalesce
+- BloomFilterMightContain
+- Boolean functions
+ - And
+ - Or
+ - Not
+ - EqualTo
+ - EqualNullSafe
+ - GreaterThan
+ - GreaterThanOrEqual
+ - LessThan
+ - LessThanOrEqual
+ - IsNull
+ - IsNotNull
+ - In
+- String functions
+ - Substring
+ - Coalesce
+ - StringSpace
+ - Like
+ - Contains
+ - Startswith
+ - Endswith
+ - Ascii
+ - Bit_length
+ - Octet_length
+ - Upper
+ - Lower
+ - Chr
+ - Initcap
+ - Trim/Btrim/Ltrim/Rtrim
+ - Concat_ws
+ - Repeat
+ - Length
+ - Reverse
+ - Instr
+ - Replace
+ - Translate
+- Bitwise functions
+ - Shiftright/Shiftleft
+- Date/Time functions
+ - Year/Hour/Minute/Second
+- Math functions
+ - Abs
+ - Acos
+ - Asin
+ - Atan
+ - Atan2
+ - Cos
+ - Exp
+ - Ln
+ - Log10
+ - Log2
+ - Pow
+ - Round
+ - Signum
+ - Sin
+ - Sqrt
+ - Tan
+ - Ceil
+ - Floor
+- Aggregate functions
+ - Count
+ - Sum
+ - Max
+ - Min
+ - Avg
+ - First
+ - Last
+ - BitAnd
+ - BitOr
+ - BitXor
+ - BoolAnd
+ - BoolOr
+ - CovPopulation
+ - CovSample
+ - VariancePop
+ - VarianceSamp
diff --git a/compatibility.html b/contributor-guide/contributing.html
similarity index 55%
copy from compatibility.html
copy to contributor-guide/contributing.html
index a01b954..a6f1e02 100644
--- a/compatibility.html
+++ b/contributor-guide/contributing.html
@@ -19,41 +19,42 @@ under the License.
<!DOCTYPE html>
-<html lang="en" data-content_root="./">
+<html lang="en" data-content_root="../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Compatibility Guide — Apache DataFusion Comet
documentation</title>
+ <title>Contributing to Apache DataFusion Comet — Apache DataFusion
Comet documentation</title>
- <link href="_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
-<link
href="_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+ <link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
<link rel="stylesheet"
- href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ href="../_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
- href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
- href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
- <link rel="stylesheet" type="text/css"
href="_static/pygments.css?v=a746c00c" />
- <link rel="stylesheet" type="text/css"
href="_static/styles/pydata-sphinx-theme.css?v=1140d252" />
- <link rel="stylesheet" type="text/css"
href="_static/theme_overrides.css?v=c6d785ac" />
+ <link rel="stylesheet" type="text/css"
href="../_static/pygments.css?v=a746c00c" />
+ <link rel="stylesheet" type="text/css"
href="../_static/styles/pydata-sphinx-theme.css?v=1140d252" />
+ <link rel="stylesheet" type="text/css"
href="../_static/theme_overrides.css?v=c6d785ac" />
- <link rel="preload" as="script"
href="_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+ <link rel="preload" as="script"
href="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
- <script src="_static/documentation_options.js?v=8a448e45"></script>
- <script src="_static/doctools.js?v=9a2dae69"></script>
- <script src="_static/sphinx_highlight.js?v=dc90522c"></script>
+ <script src="../_static/documentation_options.js?v=8a448e45"></script>
+ <script src="../_static/doctools.js?v=9a2dae69"></script>
+ <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
- <link rel="index" title="Index" href="genindex.html" />
- <link rel="search" title="Search" href="search.html" />
- <link rel="prev" title="Apache DataFusion Comet" href="index.html" />
+ <link rel="index" title="Index" href="../genindex.html" />
+ <link rel="search" title="Search" href="../search.html" />
+ <link rel="next" title="Comet Development Guide" href="development.html" />
+ <link rel="prev" title="Compatibility Guide"
href="../user-guide/compatibility.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -93,7 +94,7 @@ specific language governing permissions and limitations
under the License.
-->
-<form class="bd-search d-flex align-items-center" action="search.html"
method="get">
+<form class="bd-search d-flex align-items-center" action="../search.html"
method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input"
placeholder="Search the docs ..." aria-label="Search the docs ..."
autocomplete="off" >
</form>
@@ -103,13 +104,30 @@ under the License.
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
- Project Links
+ User Guide
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/expressions.html">
+ Supported Expressions
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/compatibility.html">
+ Compatibility Guide
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Contributor Guide
</span>
</p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
- Compatibility Guide
+ Getting Started
</a>
</li>
<li class="toctree-l1">
@@ -117,6 +135,16 @@ under the License.
Github and Issue Tracker
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="development.html">
+ Comet Development Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="debugging.html">
+ Comet Debugging Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -159,8 +187,8 @@ under the License.
</div>
- <a class="navbar-brand" href="index.html">
- <img src="_static/images/2x_bgwhite_original.png" class="logo" alt="logo">
+ <a class="navbar-brand" href="../index.html">
+ <img src="../_static/images/2x_bgwhite_original.png" class="logo"
alt="logo">
</a>
</nav>
@@ -184,13 +212,23 @@ under the License.
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#ansi-mode">
- ANSI mode
+ <a class="reference internal nav-link" href="#finding-issues-to-work-on">
+ Finding issues to work on
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#cast">
- Cast
+ <a class="reference internal nav-link" href="#reporting-issues">
+ Reporting issues
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#asking-for-help">
+ Asking for Help
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#regular-public-meetings">
+ Regular public meetings
</a>
</li>
</ul>
@@ -202,7 +240,7 @@ under the License.
<div class="tocsection editthispage">
- <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/compatibility.md";>
+ <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/contributor-guide/contributing.md";>
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -239,23 +277,34 @@ under the License.
specific language governing permissions and limitations
under the License.
-->
-<section id="compatibility-guide">
-<h1>Compatibility Guide<a class="headerlink" href="#compatibility-guide"
title="Link to this heading">¶</a></h1>
-<p>Comet aims to provide consistent results with the version of Apache Spark
that is being used.</p>
-<p>This guide offers information about areas of functionality where there are
known differences.</p>
-<section id="ansi-mode">
-<h2>ANSI mode<a class="headerlink" href="#ansi-mode" title="Link to this
heading">¶</a></h2>
-<p>Comet currently ignores ANSI mode in most cases, and therefore can produce
different results than Spark. By default,
-Comet will fall back to Spark if ANSI mode is enabled. To enable Comet to
accelerate queries when ANSI mode is enabled,
-specify <code class="docutils literal notranslate"><span
class="pre">spark.comet.ansi.enabled=true</span></code> in the Spark
configuration. Comet’s ANSI support is experimental and should not
-be used in production.</p>
-<p>There is an <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/313";>epic</a> where we
are tracking the work to fully implement ANSI support.</p>
+<section id="contributing-to-apache-datafusion-comet">
+<h1>Contributing to Apache DataFusion Comet<a class="headerlink"
href="#contributing-to-apache-datafusion-comet" title="Link to this
heading">¶</a></h1>
+<p>We welcome contributions to Comet in many areas, and encourage new
contributors to get involved.</p>
+<p>Here are some areas where you can help:</p>
+<ul class="simple">
+<li><p>Testing Comet with existing Spark jobs and reporting issues for any
bugs or performance issues</p></li>
+<li><p>Contributing code to support Spark expressions, operators, and data
types that are not currently supported</p></li>
+<li><p>Reviewing pull requests and helping to test new features for
correctness and performance</p></li>
+<li><p>Improving documentation</p></li>
+</ul>
+<section id="finding-issues-to-work-on">
+<h2>Finding issues to work on<a class="headerlink"
href="#finding-issues-to-work-on" title="Link to this heading">¶</a></h2>
+<p>We maintain a list of good first issues in GitHub <a class="reference
external"
href="https://github.com/apache/datafusion-comet/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22";>here</a>.</p>
+</section>
+<section id="reporting-issues">
+<h2>Reporting issues<a class="headerlink" href="#reporting-issues" title="Link
to this heading">¶</a></h2>
+<p>We use <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues";>GitHub issues</a> for
bug reports and feature requests.</p>
</section>
-<section id="cast">
-<h2>Cast<a class="headerlink" href="#cast" title="Link to this
heading">¶</a></h2>
-<p>Comet currently delegates to Apache DataFusion for most cast operations,
and this means that the behavior is not
-guaranteed to be consistent with Spark.</p>
-<p>There is an <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/286";>epic</a> where we
are tracking the work to implement Spark-compatible cast expressions.</p>
+<section id="asking-for-help">
+<h2>Asking for Help<a class="headerlink" href="#asking-for-help" title="Link
to this heading">¶</a></h2>
+<p>The Comet project uses the same Slack and Discord channels as the main
Apache DataFusion project. See details at
+<a class="reference external"
href="https://datafusion.apache.org/contributor-guide/communication.html";>Apache
DataFusion Communications</a>. There are dedicated Comet channels in both
Slack and Discord.</p>
+</section>
+<section id="regular-public-meetings">
+<h2>Regular public meetings<a class="headerlink"
href="#regular-public-meetings" title="Link to this heading">¶</a></h2>
+<p>The Comet contributors hold regular video calls where new and current
contributors are welcome to ask questions and
+coordinate on issues that they are working on.</p>
+<p>See the <a class="reference external"
href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing";>Apache
DataFusion Comet community meeting</a> Google document for more
information.</p>
</section>
</section>
@@ -265,13 +314,20 @@ guaranteed to be consistent with Spark.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="index.html" title="previous
page">
+ <a class='left-prev' id="prev-link"
href="../user-guide/compatibility.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Apache DataFusion Comet</p>
+ <p class="prev-next-title">Compatibility Guide</p>
</div>
</a>
+ <a class='right-next' id="next-link" href="development.html" title="next
page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">Comet Development Guide</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
</div>
</main>
@@ -280,7 +336,7 @@ guaranteed to be consistent with Spark.</p>
</div>
</div>
- <script
src="_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+ <script
src="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
<!-- Based on pydata_sphinx_theme/footer.html -->
<footer class="footer mt-5 mt-md-0">
diff --git a/contributor-guide/debugging.html b/contributor-guide/debugging.html
new file mode 100644
index 0000000..48d5e77
--- /dev/null
+++ b/contributor-guide/debugging.html
@@ -0,0 +1,476 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<!DOCTYPE html>
+
+<html lang="en" data-content_root="../">
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
+
+ <title>Comet Debugging Guide — Apache DataFusion Comet
documentation</title>
+
+ <link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+
+
+ <link rel="stylesheet"
+ href="../_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+
+
+
+
+
+ <link rel="stylesheet" type="text/css"
href="../_static/pygments.css?v=a746c00c" />
+ <link rel="stylesheet" type="text/css"
href="../_static/styles/pydata-sphinx-theme.css?v=1140d252" />
+ <link rel="stylesheet" type="text/css"
href="../_static/theme_overrides.css?v=c6d785ac" />
+
+ <link rel="preload" as="script"
href="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+
+ <script src="../_static/documentation_options.js?v=8a448e45"></script>
+ <script src="../_static/doctools.js?v=9a2dae69"></script>
+ <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
+ <script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
+ <link rel="index" title="Index" href="../genindex.html" />
+ <link rel="search" title="Search" href="../search.html" />
+ <link rel="prev" title="Comet Development Guide" href="development.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <meta name="docsearch:language" content="en">
+
+
+ <!-- Google Analytics -->
+
+ </head>
+ <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
+
+ <div class="container-fluid" id="banner"></div>
+
+
+
+
+ <div class="container-xl">
+ <div class="row">
+
+
+ <!-- Only show if we have sidebars configured, else just a small
margin -->
+ <div class="col-12 col-md-3 bd-sidebar">
+ <div class="sidebar-start-items"><!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<form class="bd-search d-flex align-items-center" action="../search.html"
method="get">
+ <i class="icon fas fa-search"></i>
+ <input type="search" class="form-control" name="q" id="search-input"
placeholder="Search the docs ..." aria-label="Search the docs ..."
autocomplete="off" >
+</form>
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ User Guide
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/expressions.html">
+ Supported Expressions
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/compatibility.html">
+ Compatibility Guide
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Contributor Guide
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributing.html">
+ Getting Started
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="development.html">
+ Comet Development Guide
+ </a>
+ </li>
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Comet Debugging Guide
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ ASF Links
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://apache.org";>
+ Apache Software Foundation
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://www.apache.org/licenses/";>
+ License
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://www.apache.org/foundation/sponsorship.html";>
+ Donate
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://www.apache.org/foundation/thanks.html";>
+ Thanks
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://www.apache.org/security/";>
+ Security
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://www.apache.org/foundation/policies/conduct.html";>
+ Code of conduct
+ </a>
+ </li>
+</ul>
+
+
+ </div>
+
+ <a class="navbar-brand" href="../index.html">
+ <img src="../_static/images/2x_bgwhite_original.png" class="logo"
alt="logo">
+ </a>
+</nav>
+
+ </div>
+ <div class="sidebar-end-items">
+ </div>
+ </div>
+
+
+
+
+ <div class="d-none d-xl-block col-xl-2 bd-toc">
+
+
+ <div class="toc-item">
+
+<div class="tocsection onthispage pt-5 pb-3">
+ <i class="fas fa-list"></i> On this page
+</div>
+
+<nav id="bd-toc-nav">
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h1 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#">
+ Comet Debugging Guide
+ </a>
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link"
href="#debugging-for-advanced-developers">
+ Debugging for Advanced Developers
+ </a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#in-intellij">
+ In Intellij
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#in-clion">
+ In CLion
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link"
href="#after-your-debugging-is-done">
+ After your debugging is done,
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#additional-info">
+ Additional Info
+ </a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="toc-h1 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#verbose-debug">
+ Verbose debug
+ </a>
+ </li>
+</ul>
+
+</nav>
+ </div>
+
+ <div class="toc-item">
+
+
+<div class="tocsection editthispage">
+ <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/contributor-guide/debugging.md";>
+ <i class="fas fa-pencil-alt"></i> Edit this page
+ </a>
+</div>
+
+ </div>
+
+
+ </div>
+
+
+
+
+
+
+ <main class="col-12 col-md-9 col-xl-7 py-md-5 pl-md-5 pr-md-4
bd-content" role="main">
+
+ <div>
+
+ <!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<section id="comet-debugging-guide">
+<h1>Comet Debugging Guide<a class="headerlink" href="#comet-debugging-guide"
title="Link to this heading">¶</a></h1>
+<p>This HOWTO describes how to debug JVM code and Native code concurrently.
The guide assumes you have:</p>
+<ol class="arabic simple">
+<li><p>Intellij as the Java IDE</p></li>
+<li><p>CLion as the Native IDE. For Rust code, the CLion Rust language plugin
is required. Note that the
+Intellij Rust plugin is not sufficient.</p></li>
+<li><p>CLion/LLDB as the native debugger. CLion ships with a bundled LLDB and
the Rust community has
+its own packaging of LLDB (<code class="docutils literal notranslate"><span
class="pre">lldb-rust</span></code>). Both provide a better display of Rust
symbols than plain
+LLDB or the LLDB that is bundled with XCode. We will use the LLDB packaged
with CLion for this guide.</p></li>
+<li><p>We will use a Comet <em>unit</em> test as the canonical use
case.</p></li>
+</ol>
+<p><em>Caveat: The steps here have only been tested with JDK 11</em> on Mac
(M1)</p>
+<section id="debugging-for-advanced-developers">
+<h2>Debugging for Advanced Developers<a class="headerlink"
href="#debugging-for-advanced-developers" title="Link to this
heading">¶</a></h2>
+<p>Add a <code class="docutils literal notranslate"><span
class="pre">.lldbinit</span></code> to comet/core. This is not strictly
necessary but will be useful if you want to
+use advanced <code class="docutils literal notranslate"><span
class="pre">lldb</span></code> debugging.</p>
+<section id="in-intellij">
+<h3>In Intellij<a class="headerlink" href="#in-intellij" title="Link to this
heading">¶</a></h3>
+<ol class="arabic">
+<li><p>Set a breakpoint in <code class="docutils literal notranslate"><span
class="pre">NativeBase.load()</span></code>, at a point <em>after</em> the
Comet library has been loaded.</p></li>
+<li><p>Add a Debug Configuration for the unit test</p></li>
+<li><p>In the Debug Configuration for that unit test add <code class="docutils
literal notranslate"><span class="pre">-Xint</span></code> as a JVM parameter.
This option is
+undocumented <em>magic</em>. Without this, the LLDB debugger hits a
EXC_BAD_ACCESS (or EXC_BAD_INSTRUCTION) from
+which one cannot recover.</p></li>
+<li><p>Add a println to the unit test to print the PID of the JVM process.
(jps can also be used but this is less error prone if you have multiple jvm
processes running)</p>
+<div class="highlight-JDK8 notranslate"><div
class="highlight"><pre><span></span> println("Waiting for Debugger:
PID - ", ManagementFactory.getRuntimeMXBean().getName())
+</pre></div>
+</div>
+<p>This will print something like : <code class="docutils literal
notranslate"><span class="pre">PID@your_machine_name</span></code>.</p>
+<p>For JDK9 and newer</p>
+<div class="highlight-JDK9 notranslate"><div
class="highlight"><pre><span></span> println("Waiting for Debugger:
PID - ", ProcessHandle.current.pid)
+</pre></div>
+</div>
+<p>==> Note the PID</p>
+</li>
+<li><p>Debug-run the test in Intellij and wait for the breakpoint to be
hit</p></li>
+</ol>
+</section>
+<section id="in-clion">
+<h3>In CLion<a class="headerlink" href="#in-clion" title="Link to this
heading">¶</a></h3>
+<ol class="arabic simple">
+<li><p>After the breakpoint is hit in Intellij, in Clion (or LLDB from
terminal or editor) -</p>
+<ol class="arabic simple">
+<li><p>Attach to the jvm process (make sure the PID matches). In CLion, this
is <code class="docutils literal notranslate"><span class="pre">Run</span>
<span class="pre">-></span> <span class="pre">Atttach</span> <span
class="pre">to</span> <span class="pre">process</span></code></p></li>
+<li><p>Put your breakpoint in the native code</p></li>
+</ol>
+</li>
+<li><p>Go back to intellij and resume the process.</p></li>
+<li><p>Most debugging in CLion is similar to Intellij. For advanced LLDB based
debugging the LLDB command line can be accessed from the LLDB tab in the
Debugger view. Refer to the <a class="reference external"
href="https://lldb.llvm.org/use/tutorial.html";>LLDB manual</a> for LLDB
commands.</p></li>
+</ol>
+</section>
+<section id="after-your-debugging-is-done">
+<h3>After your debugging is done,<a class="headerlink"
href="#after-your-debugging-is-done" title="Link to this heading">¶</a></h3>
+<ol class="arabic simple">
+<li><p>In CLion, detach from the process if not already detached</p></li>
+<li><p>In Intellij, the debugger might have lost track of the process. If so,
the debugger tab
+will show the process as running (even if the test/job is shown as
completed).</p></li>
+<li><p>Close the debugger tab, and if the IDS asks whether it should terminate
the process,
+click Yes.</p></li>
+<li><p>In terminal, use jps to identify the process with the process id you
were debugging. If
+it shows up as running, kill -9 [pid]. If that doesn’t remove the process,
don’t bother,
+the process will be left behind as a zombie and will consume no (significant)
resources.
+Eventually it will be cleaned up when you reboot possibly after a software
update.</p></li>
+</ol>
+</section>
+<section id="additional-info">
+<h3>Additional Info<a class="headerlink" href="#additional-info" title="Link
to this heading">¶</a></h3>
+<p>OpenJDK mailing list on debugging the JDK on MacOS
+https://mail.openjdk.org/pipermail/hotspot-dev/2019-September/039429.html</p>
+<p>Detecting the debugger
+https://stackoverflow.com/questions/5393403/can-a-java-application-detect-that-a-debugger-is-attached#:~:text=No.,to%20let%20your%20app%20continue.&text=I%20know%20that%20those%20are,meant%20with%20my%20first%20phrase).</p>
+</section>
+</section>
+</section>
+<section id="verbose-debug">
+<h1>Verbose debug<a class="headerlink" href="#verbose-debug" title="Link to
this heading">¶</a></h1>
+<p>By default, Comet outputs the exception details specific for Comet.</p>
+<div class="highlight-scala notranslate"><div
class="highlight"><pre><span></span><span class="n">scala</span><span
class="o">></span><span class="w"> </span><span class="n">spark</span><span
class="p">.</span><span class="n">sql</span><span class="p">(</span><span
class="s">"my_failing_query"</span><span class="p">).</span><span
class="n">show</span><span class="p">(</span><span class="kc">false</span><span
class="p">)</span>
+
+<span class="mi">24</span><span class="o">/</span><span
class="mi">03</span><span class="o">/</span><span class="mi">05</span><span
class="w"> </span><span class="mi">17</span><span class="p">:</span><span
class="mi">00</span><span class="p">:</span><span class="mi">07</span><span
class="w"> </span><span class="nc">ERROR</span><span class="w"> </span><span
class="nc">Executor</span><span class="p">:</span><span class="w"> </span><span
class="nc">Exception</span><span class="w"> </span><s [...]
+<span class="n">org</span><span class="p">.</span><span
class="n">apache</span><span class="p">.</span><span
class="n">comet</span><span class="p">.</span><span
class="nc">CometNativeException</span><span class="p">:</span><span class="w">
</span><span class="nc">Internal</span><span class="w"> </span><span
class="n">error</span><span class="p">:</span><span class="w"> </span><span
class="nc">MIN</span><span class="o">/</span><span class="nc">MAX</span><span
class="w"> </span><span class [...]
+<span class="nc">This</span><span class="w"> </span><span
class="n">was</span><span class="w"> </span><span class="n">likely</span><span
class="w"> </span><span class="n">caused</span><span class="w"> </span><span
class="n">by</span><span class="w"> </span><span class="n">a</span><span
class="w"> </span><span class="n">bug</span><span class="w"> </span><span
class="n">in</span><span class="w"> </span><span
class="nc">DataFusion</span><span class="ss">'s</span><span class="w">
</span> [...]
+<span class="w"> </span><span class="n">at</span><span class="w">
</span><span class="n">org</span><span class="p">.</span><span
class="n">apache</span><span class="p">.</span><span
class="n">comet</span><span class="p">.</span><span
class="nc">Native</span><span class="p">.</span><span
class="n">executePlan</span><span class="p">(</span><span
class="nc">Native</span><span class="w"> </span><span
class="nc">Method</span><span class="p">)</span>
+<span class="w"> </span><span class="n">at</span><span class="w">
</span><span class="n">org</span><span class="p">.</span><span
class="n">apache</span><span class="p">.</span><span
class="n">comet</span><span class="p">.</span><span
class="nc">CometExecIterator</span><span class="p">.</span><span
class="n">executeNative</span><span class="p">(</span><span
class="nc">CometExecIterator</span><span class="p">.</span><span
class="n">scala</span><span class="p">:</span><span class="mi [...]
+<span class="w"> </span><span class="n">at</span><span class="w">
</span><span class="n">org</span><span class="p">.</span><span
class="n">apache</span><span class="p">.</span><span
class="n">comet</span><span class="p">.</span><span
class="nc">CometExecIterator</span><span class="p">.</span><span
class="n">getNextBatch</span><span class="p">(</span><span
class="nc">CometExecIterator</span><span class="p">.</span><span
class="n">scala</span><span class="p">:</span><span class="mi" [...]
+<span class="w"> </span><span class="n">at</span><span class="w">
</span><span class="n">org</span><span class="p">.</span><span
class="n">apache</span><span class="p">.</span><span
class="n">comet</span><span class="p">.</span><span
class="nc">CometExecIterator</span><span class="p">.</span><span
class="n">hasNext</span><span class="p">(</span><span
class="nc">CometExecIterator</span><span class="p">.</span><span
class="n">scala</span><span class="p">:</span><span class="mi">126< [...]
+</pre></div>
+</div>
+<p>There is a verbose exception option by leveraging DataFusion <a
class="reference external"
href="https://arrow.apache.org/datafusion/user-guide/example-usage.html#enable-backtraces";>backtraces</a>
+This option allows to append native DataFusion stacktrace to the original
error message.
+To enable this option with Comet it is needed to include <code class="docutils
literal notranslate"><span class="pre">backtrace</span></code> feature in <a
class="reference external"
href="https://github.com/apache/arrow-datafusion-comet/blob/main/core/Cargo.toml";>Cargo.toml</a>
for DataFusion dependencies</p>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">datafusion</span><span
class="o">-</span><span class="n">common</span> <span class="o">=</span> <span
class="p">{</span> <span class="n">version</span> <span class="o">=</span>
<span class="s2">"36.0.0"</span><span class="p">,</span> <span
class="n">features</span> <span class="o">=</span> <span
class="p">[</span><span class="s2">"backtrace"</span><span
class="p">]</spa [...]
+<span class="n">datafusion</span> <span class="o">=</span> <span
class="p">{</span> <span class="n">default</span><span class="o">-</span><span
class="n">features</span> <span class="o">=</span> <span
class="n">false</span><span class="p">,</span> <span class="n">version</span>
<span class="o">=</span> <span class="s2">"36.0.0"</span><span
class="p">,</span> <span class="n">features</span> <span class="o">=</span>
<span class="p">[</span><span class="s2">"unicode_expressio [...]
+</pre></div>
+</div>
+<p>Then build the Comet as <a class="reference external"
href="https://github.com/apache/arrow-datafusion-comet/blob/main/README.md#getting-started";>described</a></p>
+<p>Start Comet with <code class="docutils literal notranslate"><span
class="pre">RUST_BACKTRACE=1</span></code></p>
+<div class="highlight-commandline notranslate"><div
class="highlight"><pre><span></span>RUST_BACKTRACE=1 $SPARK_HOME/spark-shell
--jars spark/target/comet-spark-spark3.4_2.12-0.1.0-SNAPSHOT.jar --conf
spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions --conf
spark.comet.enabled=true --conf spark.comet.exec.enabled=true --conf
spark.comet.exec.all.enabled=true
+</pre></div>
+</div>
+<p>Get the expanded exception details</p>
+<div class="highlight-scala notranslate"><div
class="highlight"><pre><span></span><span class="n">scala</span><span
class="o">></span><span class="w"> </span><span class="n">spark</span><span
class="p">.</span><span class="n">sql</span><span class="p">(</span><span
class="s">"my_failing_query"</span><span class="p">).</span><span
class="n">show</span><span class="p">(</span><span class="kc">false</span><span
class="p">)</span>
+<span class="mi">24</span><span class="o">/</span><span
class="mi">03</span><span class="o">/</span><span class="mi">05</span><span
class="w"> </span><span class="mi">17</span><span class="p">:</span><span
class="mi">00</span><span class="p">:</span><span class="mi">49</span><span
class="w"> </span><span class="nc">ERROR</span><span class="w"> </span><span
class="nc">Executor</span><span class="p">:</span><span class="w"> </span><span
class="nc">Exception</span><span class="w"> </span><s [...]
+<span class="n">org</span><span class="p">.</span><span
class="n">apache</span><span class="p">.</span><span
class="n">comet</span><span class="p">.</span><span
class="nc">CometNativeException</span><span class="p">:</span><span class="w">
</span><span class="nc">Internal</span><span class="w"> </span><span
class="n">error</span><span class="p">:</span><span class="w"> </span><span
class="nc">MIN</span><span class="o">/</span><span class="nc">MAX</span><span
class="w"> </span><span class [...]
+
+<span class="n">backtrace</span><span class="p">:</span>
+<span class="w"> </span><span class="mi">0</span><span
class="p">:</span><span class="w"> </span><span
class="n">std::backtrace::</span><span class="nc">Backtrace</span><span
class="n">::create</span>
+<span class="w"> </span><span class="mi">1</span><span
class="p">:</span><span class="w"> </span><span
class="n">datafusion_physical_expr::aggregate::min_max::min</span>
+<span class="w"> </span><span class="mi">2</span><span
class="p">:</span><span class="w"> </span><span class="o"><</span><span
class="n">datafusion_physical_expr::aggregate::min_max::</span><span
class="nc">MinAccumulator</span><span class="w"> </span><span
class="n">as</span><span class="w"> </span><span
class="n">datafusion_expr::accumulator::</span><span
class="nc">Accumulator</span><span class="o">>:</span><span
class="n">:update_batch</span>
+<span class="w"> </span><span class="mi">3</span><span
class="p">:</span><span class="w"> </span><span class="o"><</span><span
class="n">futures_util::stream::stream::fuse::</span><span
class="nc">Fuse</span><span class="o"><</span><span class="nc">S</span><span
class="o">></span><span class="w"> </span><span class="n">as</span><span
class="w"> </span><span class="n">futures_core::stream::</span><span
class="nc">Stream</span><span class="o">>:</span><span class="n">:poll_nex
[...]
+<span class="w"> </span><span class="mi">4</span><span
class="p">:</span><span class="w"> </span><span
class="n">comet::execution::jni_api::</span><span
class="nc">Java_org_apache_comet_Native_executePlan</span><span
class="n">::</span><span class="p">{{</span><span class="n">closure</span><span
class="p">}}</span>
+<span class="w"> </span><span class="mi">5</span><span
class="p">:</span><span class="w"> </span><span
class="n">_Java_org_apache_comet_Native_executePlan</span>
+<span class="w"> </span><span class="p">(</span><span
class="n">reduced</span><span class="p">)</span>
+
+<span class="nc">This</span><span class="w"> </span><span
class="n">was</span><span class="w"> </span><span class="n">likely</span><span
class="w"> </span><span class="n">caused</span><span class="w"> </span><span
class="n">by</span><span class="w"> </span><span class="n">a</span><span
class="w"> </span><span class="n">bug</span><span class="w"> </span><span
class="n">in</span><span class="w"> </span><span
class="nc">DataFusion</span><span class="ss">'s</span><span class="w">
</span> [...]
+<span class="w"> </span><span class="n">at</span><span class="w">
</span><span class="n">org</span><span class="p">.</span><span
class="n">apache</span><span class="p">.</span><span
class="n">comet</span><span class="p">.</span><span
class="nc">Native</span><span class="p">.</span><span
class="n">executePlan</span><span class="p">(</span><span
class="nc">Native</span><span class="w"> </span><span
class="nc">Method</span><span class="p">)</span>
+<span class="n">at</span><span class="w"> </span><span
class="n">org</span><span class="p">.</span><span class="n">apache</span><span
class="p">.</span><span class="n">comet</span><span class="p">.</span><span
class="nc">CometExecIterator</span><span class="p">.</span><span
class="n">executeNative</span><span class="p">(</span><span
class="nc">CometExecIterator</span><span class="p">.</span><span
class="n">scala</span><span class="p">:</span><span class="mi">65</span><span
class="p">)</span>
+<span class="n">at</span><span class="w"> </span><span
class="n">org</span><span class="p">.</span><span class="n">apache</span><span
class="p">.</span><span class="n">comet</span><span class="p">.</span><span
class="nc">CometExecIterator</span><span class="p">.</span><span
class="n">getNextBatch</span><span class="p">(</span><span
class="nc">CometExecIterator</span><span class="p">.</span><span
class="n">scala</span><span class="p">:</span><span class="mi">111</span><span
class="p">)</span>
+<span class="n">at</span><span class="w"> </span><span
class="n">org</span><span class="p">.</span><span class="n">apache</span><span
class="p">.</span><span class="n">comet</span><span class="p">.</span><span
class="nc">CometExecIterator</span><span class="p">.</span><span
class="n">hasNext</span><span class="p">(</span><span
class="nc">CometExecIterator</span><span class="p">.</span><span
class="n">scala</span><span class="p">:</span><span class="mi">126</span><span
class="p">)</span>
+<span class="p">(</span><span class="n">reduced</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>Note:</p>
+<ul class="simple">
+<li><p>The backtrace coverage in DataFusion is still improving. So there is a
chance the error still not covered, if so feel free to file a <a
class="reference external"
href="https://github.com/apache/arrow-datafusion/issues";>ticket</a></p></li>
+<li><p>The backtrace evaluation comes with performance cost and intended
mostly for debugging purposes</p></li>
+</ul>
+</section>
+
+
+ </div>
+
+
+ <!-- Previous / next buttons -->
+<div class='prev-next-area'>
+ <a class='left-prev' id="prev-link" href="development.html"
title="previous page">
+ <i class="fas fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">Comet Development Guide</p>
+ </div>
+ </a>
+</div>
+
+ </main>
+
+
+ </div>
+ </div>
+
+ <script
src="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+
+<!-- Based on pydata_sphinx_theme/footer.html -->
+<footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p class="copyright">
+ © Copyright 2023-2024, Apache Software Foundation.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p class="sphinx-version">
+Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 7.3.7.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p>Apache DataFusion, Apache DataFusion Comet, Apache, the Apache
feather logo, and the Apache DataFusion project logo</p>
+ <p>are either registered trademarks or trademarks of The Apache Software
Foundation in the United States and other countries.</p>
+ </div>
+ </div>
+</footer>
+
+
+ </body>
+</html>
\ No newline at end of file
diff --git a/contributor-guide/development.html
b/contributor-guide/development.html
new file mode 100644
index 0000000..7b27461
--- /dev/null
+++ b/contributor-guide/development.html
@@ -0,0 +1,447 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<!DOCTYPE html>
+
+<html lang="en" data-content_root="../">
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
+
+ <title>Comet Development Guide — Apache DataFusion Comet
documentation</title>
+
+ <link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+
+
+ <link rel="stylesheet"
+ href="../_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+
+
+
+
+
+ <link rel="stylesheet" type="text/css"
href="../_static/pygments.css?v=a746c00c" />
+ <link rel="stylesheet" type="text/css"
href="../_static/styles/pydata-sphinx-theme.css?v=1140d252" />
+ <link rel="stylesheet" type="text/css"
href="../_static/theme_overrides.css?v=c6d785ac" />
+
+ <link rel="preload" as="script"
href="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+
+ <script src="../_static/documentation_options.js?v=8a448e45"></script>
+ <script src="../_static/doctools.js?v=9a2dae69"></script>
+ <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
+ <script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
+ <link rel="index" title="Index" href="../genindex.html" />
+ <link rel="search" title="Search" href="../search.html" />
+ <link rel="next" title="Comet Debugging Guide" href="debugging.html" />
+ <link rel="prev" title="Contributing to Apache DataFusion Comet"
href="contributing.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <meta name="docsearch:language" content="en">
+
+
+ <!-- Google Analytics -->
+
+ </head>
+ <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
+
+ <div class="container-fluid" id="banner"></div>
+
+
+
+
+ <div class="container-xl">
+ <div class="row">
+
+
+ <!-- Only show if we have sidebars configured, else just a small
margin -->
+ <div class="col-12 col-md-3 bd-sidebar">
+ <div class="sidebar-start-items"><!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<form class="bd-search d-flex align-items-center" action="../search.html"
method="get">
+ <i class="icon fas fa-search"></i>
+ <input type="search" class="form-control" name="q" id="search-input"
placeholder="Search the docs ..." aria-label="Search the docs ..."
autocomplete="off" >
+</form>
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ User Guide
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/expressions.html">
+ Supported Expressions
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/compatibility.html">
+ Compatibility Guide
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Contributor Guide
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributing.html">
+ Getting Started
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Comet Development Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="debugging.html">
+ Comet Debugging Guide
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ ASF Links
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://apache.org";>
+ Apache Software Foundation
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://www.apache.org/licenses/";>
+ License
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://www.apache.org/foundation/sponsorship.html";>
+ Donate
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://www.apache.org/foundation/thanks.html";>
+ Thanks
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://www.apache.org/security/";>
+ Security
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://www.apache.org/foundation/policies/conduct.html";>
+ Code of conduct
+ </a>
+ </li>
+</ul>
+
+
+ </div>
+
+ <a class="navbar-brand" href="../index.html">
+ <img src="../_static/images/2x_bgwhite_original.png" class="logo"
alt="logo">
+ </a>
+</nav>
+
+ </div>
+ <div class="sidebar-end-items">
+ </div>
+ </div>
+
+
+
+
+ <div class="d-none d-xl-block col-xl-2 bd-toc">
+
+
+ <div class="toc-item">
+
+<div class="tocsection onthispage pt-5 pb-3">
+ <i class="fas fa-list"></i> On this page
+</div>
+
+<nav id="bd-toc-nav">
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#project-layout">
+ Project Layout
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#development-setup">
+ Development Setup
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#build-test">
+ Build & Test
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#development-environment">
+ Development Environment
+ </a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#intellij-idea">
+ IntelliJ IDEA
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#clion">
+ CLion
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#running-tests-in-idea">
+ Running Tests in IDEA
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#benchmark">
+ Benchmark
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#debugging">
+ Debugging
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#submitting-a-pull-request">
+ Submitting a Pull Request
+ </a>
+ </li>
+</ul>
+
+</nav>
+ </div>
+
+ <div class="toc-item">
+
+
+<div class="tocsection editthispage">
+ <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/contributor-guide/development.md";>
+ <i class="fas fa-pencil-alt"></i> Edit this page
+ </a>
+</div>
+
+ </div>
+
+
+ </div>
+
+
+
+
+
+
+ <main class="col-12 col-md-9 col-xl-7 py-md-5 pl-md-5 pr-md-4
bd-content" role="main">
+
+ <div>
+
+ <!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<section id="comet-development-guide">
+<h1>Comet Development Guide<a class="headerlink"
href="#comet-development-guide" title="Link to this heading">¶</a></h1>
+<section id="project-layout">
+<h2>Project Layout<a class="headerlink" href="#project-layout" title="Link to
this heading">¶</a></h2>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span>├── common <- common Java/Scala code
+├── conf <- configuration files
+├── core <- core native code, in Rust
+├── spark <- Spark integration
+</pre></div>
+</div>
+</section>
+<section id="development-setup">
+<h2>Development Setup<a class="headerlink" href="#development-setup"
title="Link to this heading">¶</a></h2>
+<ol class="arabic simple">
+<li><p>Make sure <code class="docutils literal notranslate"><span
class="pre">JAVA_HOME</span></code> is set and point to JDK 8/11/17
installation.</p></li>
+<li><p>Install Rust toolchain. The easiest way is to use
+<a class="reference external" href="https://rustup.rs";>rustup</a>.</p></li>
+</ol>
+</section>
+<section id="build-test">
+<h2>Build & Test<a class="headerlink" href="#build-test" title="Link to
this heading">¶</a></h2>
+<p>A few common commands are specified in project’s <code class="docutils
literal notranslate"><span class="pre">Makefile</span></code>:</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span
class="pre">make</span></code>: compile the entire project, but don’t run
tests</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">make</span> <span class="pre">test-rust</span></code>: compile the
project and run tests in Rust side</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">make</span> <span class="pre">test-java</span></code>: compile the
project and run tests in Java side</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">make</span> <span class="pre">test</span></code>: compile the
project and run tests in both Rust and Java
+side.</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">make</span> <span class="pre">release</span></code>: compile the
project and creates a release build. This
+is useful when you want to test Comet local installation in another project
+such as Spark.</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">make</span> <span class="pre">clean</span></code>: clean up the
workspace</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">bin/comet-spark-shell</span> <span class="pre">-d</span> <span
class="pre">.</span> <span class="pre">-o</span> <span
class="pre">spark/target/</span></code> run Comet spark shell for V1
datasources</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">bin/comet-spark-shell</span> <span class="pre">-d</span> <span
class="pre">.</span> <span class="pre">-o</span> <span
class="pre">spark/target/</span> <span class="pre">--conf</span> <span
class="pre">spark.sql.sources.useV1SourceList=""</span></code> run
Comet spark shell for V2 datasources</p></li>
+</ul>
+</section>
+<section id="development-environment">
+<h2>Development Environment<a class="headerlink"
href="#development-environment" title="Link to this heading">¶</a></h2>
+<p>Comet is a multi-language project with native code written in Rust and JVM
code written in Java and Scala.
+For Rust code, the CLion IDE is recommended. For JVM code, IntelliJ IDEA is
recommended.</p>
+<p>Before opening the project in an IDE, make sure to run <code
class="docutils literal notranslate"><span class="pre">make</span></code> first
to generate the necessary files for the IDEs. Currently, it’s mostly about
+generating protobuf message classes for the JVM side. It’s only required to
run <code class="docutils literal notranslate"><span
class="pre">make</span></code> once after cloning the repo.</p>
+<section id="intellij-idea">
+<h3>IntelliJ IDEA<a class="headerlink" href="#intellij-idea" title="Link to
this heading">¶</a></h3>
+<p>First make sure to install the Scala plugin in IntelliJ IDEA.
+After that, you can open the project in IntelliJ IDEA. The IDE should
automatically detect the project structure and import as a Maven project.</p>
+</section>
+<section id="clion">
+<h3>CLion<a class="headerlink" href="#clion" title="Link to this
heading">¶</a></h3>
+<p>First make sure to install the Rust plugin in CLion or you can use the
dedicated Rust IDE: RustRover.
+After that you can open the project in CLion. The IDE should automatically
detect the project structure and import as a Cargo project.</p>
+</section>
+<section id="running-tests-in-idea">
+<h3>Running Tests in IDEA<a class="headerlink" href="#running-tests-in-idea"
title="Link to this heading">¶</a></h3>
+<p>Like other Maven projects, you can run tests in IntelliJ IDEA by
right-clicking on the test class or test method and selecting “Run” or “Debug”.
+However if the tests is related to the native side. Please make sure to run
<code class="docutils literal notranslate"><span class="pre">make</span> <span
class="pre">core</span></code> or <code class="docutils literal
notranslate"><span class="pre">cd</span> <span class="pre">core</span> <span
class="pre">&&</span> <span class="pre">cargo</span> <span
class="pre">build</span></code> before running the tests in IDEA.</p>
+</section>
+</section>
+<section id="benchmark">
+<h2>Benchmark<a class="headerlink" href="#benchmark" title="Link to this
heading">¶</a></h2>
+<p>There’s a <code class="docutils literal notranslate"><span
class="pre">make</span></code> command to run micro benchmarks in the repo. For
+instance:</p>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">make</span> <span
class="n">benchmark</span><span class="o">-</span><span
class="n">org</span><span class="o">.</span><span class="n">apache</span><span
class="o">.</span><span class="n">spark</span><span class="o">.</span><span
class="n">sql</span><span class="o">.</span><span
class="n">benchmark</span><span class="o">.</span><span
class="n">CometReadBenchmark</span>
+</pre></div>
+</div>
+<p>To run TPC-H or TPC-DS micro benchmarks, please follow the instructions
+in the respective source code, e.g., <code class="docutils literal
notranslate"><span class="pre">CometTPCHQueryBenchmark</span></code>.</p>
+</section>
+<section id="debugging">
+<h2>Debugging<a class="headerlink" href="#debugging" title="Link to this
heading">¶</a></h2>
+<p>Comet is a multi-language project with native code written in Rust and JVM
code written in Java and Scala.
+It is possible to debug both native and JVM code concurrently as described in
the <a class="reference internal" href="debugging.html"><span class="doc std
std-doc">DEBUGGING guide</span></a></p>
+</section>
+<section id="submitting-a-pull-request">
+<h2>Submitting a Pull Request<a class="headerlink"
href="#submitting-a-pull-request" title="Link to this heading">¶</a></h2>
+<p>Comet uses <code class="docutils literal notranslate"><span
class="pre">cargo</span> <span class="pre">fmt</span></code>, <a
class="reference external"
href="https://github.com/scalacenter/scalafix";>Scalafix</a> and <a
class="reference external"
href="https://github.com/diffplug/spotless/tree/main/plugin-maven";>Spotless</a>
to
+automatically format the code. Before submitting a pull request, you can
simply run <code class="docutils literal notranslate"><span
class="pre">make</span> <span class="pre">format</span></code> to format the
code.</p>
+</section>
+</section>
+
+
+ </div>
+
+
+ <!-- Previous / next buttons -->
+<div class='prev-next-area'>
+ <a class='left-prev' id="prev-link" href="contributing.html"
title="previous page">
+ <i class="fas fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">Contributing to Apache DataFusion
Comet</p>
+ </div>
+ </a>
+ <a class='right-next' id="next-link" href="debugging.html" title="next
page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">Comet Debugging Guide</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
+</div>
+
+ </main>
+
+
+ </div>
+ </div>
+
+ <script
src="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+
+<!-- Based on pydata_sphinx_theme/footer.html -->
+<footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p class="copyright">
+ © Copyright 2023-2024, Apache Software Foundation.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p class="sphinx-version">
+Created using <a href="http://sphinx-doc.org/";>Sphinx</a> 7.3.7.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p>Apache DataFusion, Apache DataFusion Comet, Apache, the Apache
feather logo, and the Apache DataFusion project logo</p>
+ <p>are either registered trademarks or trademarks of The Apache Software
Foundation in the United States and other countries.</p>
+ </div>
+ </div>
+</footer>
+
+
+ </body>
+</html>
\ No newline at end of file
diff --git a/genindex.html b/genindex.html
index c2c1c54..8a16123 100644
--- a/genindex.html
+++ b/genindex.html
@@ -101,20 +101,47 @@ under the License.
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
- Project Links
+ User Guide
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
- <a class="reference internal" href="compatibility.html">
+ <a class="reference internal" href="user-guide/expressions.html">
+ Supported Expressions
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/compatibility.html">
Compatibility Guide
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Contributor Guide
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/contributing.html">
+ Getting Started
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/development.html">
+ Comet Development Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/debugging.html">
+ Comet Debugging Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/index.html b/index.html
index c5b34f9..cc8a388 100644
--- a/index.html
+++ b/index.html
@@ -53,7 +53,7 @@ under the License.
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
- <link rel="next" title="Compatibility Guide" href="compatibility.html" />
+ <link rel="next" title="Supported Spark Expressions"
href="user-guide/expressions.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -103,20 +103,47 @@ under the License.
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
- Project Links
+ User Guide
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
- <a class="reference internal" href="compatibility.html">
+ <a class="reference internal" href="user-guide/expressions.html">
+ Supported Expressions
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/compatibility.html">
Compatibility Guide
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Contributor Guide
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/contributing.html">
+ Getting Started
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/development.html">
+ Comet Development Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/debugging.html">
+ Comet Debugging Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -216,13 +243,21 @@ under the License.
<a class="github-button"
href="https://github.com/apache/datafusion-comet/fork"; data-size="large"
data-show-count="true" aria-label="Fork apache/datafusion-comet on
GitHub">Fork</a>
</p><p>Apache DataFusion Comet is an Apache Spark plugin that uses Apache
DataFusion
as a native runtime to achieve improvement in terms of query efficiency and
query runtime.</p>
-<p>This documentation site is currently being developed. The most up-to-date
documentation can be found in the
-GitHub repository at <a class="reference external"
href="https://github.com/apache/datafusion-comet";>https://github.com/apache/datafusion-comet</a>.</p>
<div class="toctree-wrapper compound" id="toc-links">
-<p aria-level="2" class="caption" role="heading"><span
class="caption-text">Project Links</span></p>
+<p aria-level="2" class="caption" role="heading"><span
class="caption-text">User Guide</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal"
href="user-guide/expressions.html">Supported Expressions</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="user-guide/compatibility.html">Compatibility Guide</a></li>
+</ul>
+</div>
+<div class="toctree-wrapper compound" id="id1">
+<p aria-level="2" class="caption" role="heading"><span
class="caption-text">Contributor Guide</span></p>
<ul>
-<li class="toctree-l1"><a class="reference internal"
href="compatibility.html">Compatibility Guide</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/contributing.html">Getting Started</a></li>
<li class="toctree-l1"><a class="reference external"
href="https://github.com/apache/datafusion-comet";>Github and Issue
Tracker</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/development.html">Comet Development Guide</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/debugging.html">Comet Debugging Guide</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/debugging.html#verbose-debug">Verbose debug</a></li>
</ul>
</div>
<div class="toctree-wrapper compound" id="toc-asf-links">
@@ -244,10 +279,10 @@ GitHub repository at <a class="reference external"
href="https://github.com/apac
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='right-next' id="next-link" href="compatibility.html" title="next
page">
+ <a class='right-next' id="next-link" href="user-guide/expressions.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Compatibility Guide</p>
+ <p class="prev-next-title">Supported Spark Expressions</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/objects.inv b/objects.inv
index d1dd80f..fc585e9 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/search.html b/search.html
index eb4c545..7939299 100644
--- a/search.html
+++ b/search.html
@@ -108,20 +108,47 @@ under the License.
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
- Project Links
+ User Guide
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
- <a class="reference internal" href="compatibility.html">
+ <a class="reference internal" href="user-guide/expressions.html">
+ Supported Expressions
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/compatibility.html">
Compatibility Guide
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Contributor Guide
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/contributing.html">
+ Getting Started
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/development.html">
+ Comet Development Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/debugging.html">
+ Comet Debugging Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/searchindex.js b/searchindex.js
index 370a835..4f796f8 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"ANSI mode": [[0, "ansi-mode"]], "ASF Links":
[[1, null]], "Apache DataFusion Comet": [[1, "apache-datafusion-comet"]],
"Cast": [[0, "cast"]], "Compatibility Guide": [[0, "compatibility-guide"]],
"Project Links": [[1, null]]}, "docnames": ["compatibility", "index"],
"envversion": {"sphinx": 61, "sphinx.domains.c": 3, "sphinx.domains.changeset":
1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9,
"sphinx.domains.index": 1, "sphinx.domains.javascript": [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"ANSI mode": [[4, "ansi-mode"]], "ASF Links":
[[3, null]], "Additional Info": [[1, "additional-info"]], "After your debugging
is done,": [[1, "after-your-debugging-is-done"]], "Apache DataFusion Comet":
[[3, "apache-datafusion-comet"]], "Asking for Help": [[0, "asking-for-help"]],
"Benchmark": [[2, "benchmark"]], "Build & Test": [[2, "build-test"]], "CLion":
[[2, "clion"]], "Cast": [[4, "cast"]], "Comet Debugging Guide": [[1,
"comet-debugging-guide"]], "Com [...]
\ No newline at end of file
diff --git a/compatibility.html b/user-guide/compatibility.html
similarity index 75%
copy from compatibility.html
copy to user-guide/compatibility.html
index a01b954..c8a6aa6 100644
--- a/compatibility.html
+++ b/user-guide/compatibility.html
@@ -19,41 +19,42 @@ under the License.
<!DOCTYPE html>
-<html lang="en" data-content_root="./">
+<html lang="en" data-content_root="../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Compatibility Guide — Apache DataFusion Comet
documentation</title>
- <link href="_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
-<link
href="_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+ <link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
<link rel="stylesheet"
- href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ href="../_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
- href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
- href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
- <link rel="stylesheet" type="text/css"
href="_static/pygments.css?v=a746c00c" />
- <link rel="stylesheet" type="text/css"
href="_static/styles/pydata-sphinx-theme.css?v=1140d252" />
- <link rel="stylesheet" type="text/css"
href="_static/theme_overrides.css?v=c6d785ac" />
+ <link rel="stylesheet" type="text/css"
href="../_static/pygments.css?v=a746c00c" />
+ <link rel="stylesheet" type="text/css"
href="../_static/styles/pydata-sphinx-theme.css?v=1140d252" />
+ <link rel="stylesheet" type="text/css"
href="../_static/theme_overrides.css?v=c6d785ac" />
- <link rel="preload" as="script"
href="_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+ <link rel="preload" as="script"
href="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
- <script src="_static/documentation_options.js?v=8a448e45"></script>
- <script src="_static/doctools.js?v=9a2dae69"></script>
- <script src="_static/sphinx_highlight.js?v=dc90522c"></script>
+ <script src="../_static/documentation_options.js?v=8a448e45"></script>
+ <script src="../_static/doctools.js?v=9a2dae69"></script>
+ <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
- <link rel="index" title="Index" href="genindex.html" />
- <link rel="search" title="Search" href="search.html" />
- <link rel="prev" title="Apache DataFusion Comet" href="index.html" />
+ <link rel="index" title="Index" href="../genindex.html" />
+ <link rel="search" title="Search" href="../search.html" />
+ <link rel="next" title="Contributing to Apache DataFusion Comet"
href="../contributor-guide/contributing.html" />
+ <link rel="prev" title="Supported Spark Expressions"
href="expressions.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -93,7 +94,7 @@ specific language governing permissions and limitations
under the License.
-->
-<form class="bd-search d-flex align-items-center" action="search.html"
method="get">
+<form class="bd-search d-flex align-items-center" action="../search.html"
method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input"
placeholder="Search the docs ..." aria-label="Search the docs ..."
autocomplete="off" >
</form>
@@ -103,20 +104,47 @@ under the License.
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
- Project Links
+ User Guide
</span>
</p>
<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="expressions.html">
+ Supported Expressions
+ </a>
+ </li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
Compatibility Guide
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Contributor Guide
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/contributing.html">
+ Getting Started
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/development.html">
+ Comet Development Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/debugging.html">
+ Comet Debugging Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -159,8 +187,8 @@ under the License.
</div>
- <a class="navbar-brand" href="index.html">
- <img src="_static/images/2x_bgwhite_original.png" class="logo" alt="logo">
+ <a class="navbar-brand" href="../index.html">
+ <img src="../_static/images/2x_bgwhite_original.png" class="logo"
alt="logo">
</a>
</nav>
@@ -202,7 +230,7 @@ under the License.
<div class="tocsection editthispage">
- <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/compatibility.md";>
+ <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/user-guide/compatibility.md";>
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -265,13 +293,20 @@ guaranteed to be consistent with Spark.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="index.html" title="previous
page">
+ <a class='left-prev' id="prev-link" href="expressions.html"
title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Apache DataFusion Comet</p>
+ <p class="prev-next-title">Supported Spark Expressions</p>
</div>
</a>
+ <a class='right-next' id="next-link"
href="../contributor-guide/contributing.html" title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">Contributing to Apache DataFusion Comet</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
</div>
</main>
@@ -280,7 +315,7 @@ guaranteed to be consistent with Spark.</p>
</div>
</div>
- <script
src="_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+ <script
src="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
<!-- Based on pydata_sphinx_theme/footer.html -->
<footer class="footer mt-5 mt-md-0">
diff --git a/compatibility.html b/user-guide/expressions.html
similarity index 57%
rename from compatibility.html
rename to user-guide/expressions.html
index a01b954..b6d8f51 100644
--- a/compatibility.html
+++ b/user-guide/expressions.html
@@ -19,41 +19,42 @@ under the License.
<!DOCTYPE html>
-<html lang="en" data-content_root="./">
+<html lang="en" data-content_root="../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Compatibility Guide — Apache DataFusion Comet
documentation</title>
+ <title>Supported Spark Expressions — Apache DataFusion Comet
documentation</title>
- <link href="_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
-<link
href="_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+ <link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
+<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
<link rel="stylesheet"
- href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ href="../_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
- href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
- href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
- <link rel="stylesheet" type="text/css"
href="_static/pygments.css?v=a746c00c" />
- <link rel="stylesheet" type="text/css"
href="_static/styles/pydata-sphinx-theme.css?v=1140d252" />
- <link rel="stylesheet" type="text/css"
href="_static/theme_overrides.css?v=c6d785ac" />
+ <link rel="stylesheet" type="text/css"
href="../_static/pygments.css?v=a746c00c" />
+ <link rel="stylesheet" type="text/css"
href="../_static/styles/pydata-sphinx-theme.css?v=1140d252" />
+ <link rel="stylesheet" type="text/css"
href="../_static/theme_overrides.css?v=c6d785ac" />
- <link rel="preload" as="script"
href="_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+ <link rel="preload" as="script"
href="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
- <script src="_static/documentation_options.js?v=8a448e45"></script>
- <script src="_static/doctools.js?v=9a2dae69"></script>
- <script src="_static/sphinx_highlight.js?v=dc90522c"></script>
+ <script src="../_static/documentation_options.js?v=8a448e45"></script>
+ <script src="../_static/doctools.js?v=9a2dae69"></script>
+ <script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
- <link rel="index" title="Index" href="genindex.html" />
- <link rel="search" title="Search" href="search.html" />
- <link rel="prev" title="Apache DataFusion Comet" href="index.html" />
+ <link rel="index" title="Index" href="../genindex.html" />
+ <link rel="search" title="Search" href="../search.html" />
+ <link rel="next" title="Compatibility Guide" href="compatibility.html" />
+ <link rel="prev" title="Apache DataFusion Comet" href="../index.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -93,7 +94,7 @@ specific language governing permissions and limitations
under the License.
-->
-<form class="bd-search d-flex align-items-center" action="search.html"
method="get">
+<form class="bd-search d-flex align-items-center" action="../search.html"
method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input"
placeholder="Search the docs ..." aria-label="Search the docs ..."
autocomplete="off" >
</form>
@@ -103,20 +104,47 @@ under the License.
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
- Project Links
+ User Guide
</span>
</p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
+ Supported Expressions
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="compatibility.html">
Compatibility Guide
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Contributor Guide
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/contributing.html">
+ Getting Started
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/development.html">
+ Comet Development Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/debugging.html">
+ Comet Debugging Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -159,8 +187,8 @@ under the License.
</div>
- <a class="navbar-brand" href="index.html">
- <img src="_static/images/2x_bgwhite_original.png" class="logo" alt="logo">
+ <a class="navbar-brand" href="../index.html">
+ <img src="../_static/images/2x_bgwhite_original.png" class="logo"
alt="logo">
</a>
</nav>
@@ -177,24 +205,9 @@ under the License.
<div class="toc-item">
-<div class="tocsection onthispage pt-5 pb-3">
- <i class="fas fa-list"></i> On this page
-</div>
<nav id="bd-toc-nav">
- <ul class="visible nav section-nav flex-column">
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#ansi-mode">
- ANSI mode
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#cast">
- Cast
- </a>
- </li>
-</ul>
-
+
</nav>
</div>
@@ -202,7 +215,7 @@ under the License.
<div class="tocsection editthispage">
- <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/compatibility.md";>
+ <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/user-guide/expressions.md";>
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -239,24 +252,121 @@ under the License.
specific language governing permissions and limitations
under the License.
-->
-<section id="compatibility-guide">
-<h1>Compatibility Guide<a class="headerlink" href="#compatibility-guide"
title="Link to this heading">¶</a></h1>
-<p>Comet aims to provide consistent results with the version of Apache Spark
that is being used.</p>
-<p>This guide offers information about areas of functionality where there are
known differences.</p>
-<section id="ansi-mode">
-<h2>ANSI mode<a class="headerlink" href="#ansi-mode" title="Link to this
heading">¶</a></h2>
-<p>Comet currently ignores ANSI mode in most cases, and therefore can produce
different results than Spark. By default,
-Comet will fall back to Spark if ANSI mode is enabled. To enable Comet to
accelerate queries when ANSI mode is enabled,
-specify <code class="docutils literal notranslate"><span
class="pre">spark.comet.ansi.enabled=true</span></code> in the Spark
configuration. Comet’s ANSI support is experimental and should not
-be used in production.</p>
-<p>There is an <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/313";>epic</a> where we
are tracking the work to fully implement ANSI support.</p>
-</section>
-<section id="cast">
-<h2>Cast<a class="headerlink" href="#cast" title="Link to this
heading">¶</a></h2>
-<p>Comet currently delegates to Apache DataFusion for most cast operations,
and this means that the behavior is not
-guaranteed to be consistent with Spark.</p>
-<p>There is an <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/286";>epic</a> where we
are tracking the work to implement Spark-compatible cast expressions.</p>
-</section>
+<section id="supported-spark-expressions">
+<h1>Supported Spark Expressions<a class="headerlink"
href="#supported-spark-expressions" title="Link to this heading">¶</a></h1>
+<p>The following Spark expressions are currently available:</p>
+<ul class="simple">
+<li><p>Literals</p></li>
+<li><p>Arithmetic Operators</p>
+<ul>
+<li><p>UnaryMinus</p></li>
+<li><p>Add/Minus/Multiply/Divide/Remainder</p></li>
+</ul>
+</li>
+<li><p>Conditional functions</p>
+<ul>
+<li><p>Case When</p></li>
+<li><p>If</p></li>
+</ul>
+</li>
+<li><p>Cast</p></li>
+<li><p>Coalesce</p></li>
+<li><p>BloomFilterMightContain</p></li>
+<li><p>Boolean functions</p>
+<ul>
+<li><p>And</p></li>
+<li><p>Or</p></li>
+<li><p>Not</p></li>
+<li><p>EqualTo</p></li>
+<li><p>EqualNullSafe</p></li>
+<li><p>GreaterThan</p></li>
+<li><p>GreaterThanOrEqual</p></li>
+<li><p>LessThan</p></li>
+<li><p>LessThanOrEqual</p></li>
+<li><p>IsNull</p></li>
+<li><p>IsNotNull</p></li>
+<li><p>In</p></li>
+</ul>
+</li>
+<li><p>String functions</p>
+<ul>
+<li><p>Substring</p></li>
+<li><p>Coalesce</p></li>
+<li><p>StringSpace</p></li>
+<li><p>Like</p></li>
+<li><p>Contains</p></li>
+<li><p>Startswith</p></li>
+<li><p>Endswith</p></li>
+<li><p>Ascii</p></li>
+<li><p>Bit_length</p></li>
+<li><p>Octet_length</p></li>
+<li><p>Upper</p></li>
+<li><p>Lower</p></li>
+<li><p>Chr</p></li>
+<li><p>Initcap</p></li>
+<li><p>Trim/Btrim/Ltrim/Rtrim</p></li>
+<li><p>Concat_ws</p></li>
+<li><p>Repeat</p></li>
+<li><p>Length</p></li>
+<li><p>Reverse</p></li>
+<li><p>Instr</p></li>
+<li><p>Replace</p></li>
+<li><p>Translate</p></li>
+</ul>
+</li>
+<li><p>Bitwise functions</p>
+<ul>
+<li><p>Shiftright/Shiftleft</p></li>
+</ul>
+</li>
+<li><p>Date/Time functions</p>
+<ul>
+<li><p>Year/Hour/Minute/Second</p></li>
+</ul>
+</li>
+<li><p>Math functions</p>
+<ul>
+<li><p>Abs</p></li>
+<li><p>Acos</p></li>
+<li><p>Asin</p></li>
+<li><p>Atan</p></li>
+<li><p>Atan2</p></li>
+<li><p>Cos</p></li>
+<li><p>Exp</p></li>
+<li><p>Ln</p></li>
+<li><p>Log10</p></li>
+<li><p>Log2</p></li>
+<li><p>Pow</p></li>
+<li><p>Round</p></li>
+<li><p>Signum</p></li>
+<li><p>Sin</p></li>
+<li><p>Sqrt</p></li>
+<li><p>Tan</p></li>
+<li><p>Ceil</p></li>
+<li><p>Floor</p></li>
+</ul>
+</li>
+<li><p>Aggregate functions</p>
+<ul>
+<li><p>Count</p></li>
+<li><p>Sum</p></li>
+<li><p>Max</p></li>
+<li><p>Min</p></li>
+<li><p>Avg</p></li>
+<li><p>First</p></li>
+<li><p>Last</p></li>
+<li><p>BitAnd</p></li>
+<li><p>BitOr</p></li>
+<li><p>BitXor</p></li>
+<li><p>BoolAnd</p></li>
+<li><p>BoolOr</p></li>
+<li><p>CovPopulation</p></li>
+<li><p>CovSample</p></li>
+<li><p>VariancePop</p></li>
+<li><p>VarianceSamp</p></li>
+</ul>
+</li>
+</ul>
</section>
@@ -265,13 +375,20 @@ guaranteed to be consistent with Spark.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="index.html" title="previous
page">
+ <a class='left-prev' id="prev-link" href="../index.html" title="previous
page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
<p class="prev-next-title">Apache DataFusion Comet</p>
</div>
</a>
+ <a class='right-next' id="next-link" href="compatibility.html" title="next
page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">Compatibility Guide</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
</div>
</main>
@@ -280,7 +397,7 @@ guaranteed to be consistent with Spark.</p>
</div>
</div>
- <script
src="_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+ <script
src="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
<!-- Based on pydata_sphinx_theme/footer.html -->
<footer class="footer mt-5 mt-md-0">
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
