This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 62cb6236aa Publish built docs triggered by
936794d7f6884e4a71665d1c4390b8877a43a7ee
62cb6236aa is described below
commit 62cb6236aae4ab571e293e289e9a9f101f7545e7
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon Aug 3 14:49:26 2026 +0000
Publish built docs triggered by 936794d7f6884e4a71665d1c4390b8877a43a7ee
---
.../contributor-guide/config_conventions.md.txt | 27 +++++++++++-----------
_sources/user-guide/latest/configs.md.txt | 2 +-
_sources/user-guide/latest/pyarrow-udfs.md.txt | 8 +++----
contributor-guide/config_conventions.html | 19 +++++++--------
searchindex.js | 2 +-
user-guide/latest/configs.html | 2 +-
user-guide/latest/pyarrow-udfs.html | 8 +++----
7 files changed, 35 insertions(+), 33 deletions(-)
diff --git a/_sources/contributor-guide/config_conventions.md.txt
b/_sources/contributor-guide/config_conventions.md.txt
index 90042a71c0..5bcad087ea 100644
--- a/_sources/contributor-guide/config_conventions.md.txt
+++ b/_sources/contributor-guide/config_conventions.md.txt
@@ -34,9 +34,9 @@ broadest (the prefix) to narrowest (the specific setting).
prefix like `spark.sql.comet.`.
- **Category segment.** The first segment after the prefix names a broad area:
`exec` (execution and expressions), `scan` (source readers), `parquet`
(Parquet-specific
- settings), `shuffle` (shuffle behavior), `columnar` (columnar shuffle),
`explain` (plan
- explain/logging), `metrics`, `tracing`, `testing`, `convert` (Spark → Comet
source
- conversion), or `expression` / `operator` (per-expression / per-operator
flags).
+ settings), `shuffle` (shuffle behavior), `explain` (plan explain/logging),
`metrics`,
+ `tracing`, `debug`, `testing`, `convert` (Spark → Comet source conversion),
or
+ `expression` / `operator` (per-expression / per-operator flags).
- **Segment casing.** Multi-word segments use `camelCase`, not dot- or
kebab-separated.
Prefer `spark.comet.foo.maxThreadNum` over `spark.comet.foo.max.thread.num`
or
`spark.comet.foo.max-thread-num`.
@@ -46,8 +46,8 @@ broadest (the prefix) to narrowest (the specific setting).
a verb (`force...`, `allow...`) can omit `.enabled` because the verb already
encodes
the action — for example `spark.comet.exec.forceShuffledHashJoin`.
- **Acronyms.** Treat acronyms consistently as `UDF`, `SHJ`, `SMJ`, `IO` —
either all-caps
- or camelCase, but do not mix within one key (`pyarrowUdf` and `scalaUDF` in
the same
- cluster is a red flag).
+ or camelCase, but do not mix within one cluster (`pyarrowUdf` alongside
`scalaUDF` would
+ be a red flag; both spell it `UDF`).
## Symbol Naming (Scala)
@@ -56,11 +56,11 @@ Every config declares a Scala `val` in `CometConf.scala`.
The symbol name is the
Examples:
-| Key | Symbol
|
-| ----------------------------------------------- |
--------------------------- |
-| `spark.comet.enabled` | `COMET_ENABLED`
|
-| `spark.comet.exec.forceShuffledHashJoin` | `COMET_FORCE_SHJ`
|
-| `spark.comet.parquet.rowFilterPushdown.enabled` |
`COMET_ROW_FILTER_PUSHDOWN` |
+| Key | Symbol
|
+| ----------------------------------------------- |
------------------------------------------- |
+| `spark.comet.enabled` | `COMET_ENABLED`
|
+| `spark.comet.exec.forceShuffledHashJoin` | `COMET_FORCE_SHJ`
|
+| `spark.comet.parquet.rowFilterPushdown.enabled` |
`COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED` |
The symbol name is what appears in code; the key is what appears in user
configuration.
The two do not have to match segment-for-segment — brevity in the symbol is
fine as long
@@ -137,9 +137,10 @@ val COMET_LEGACY_FOO_BEHAVIOR: ConfigEntry[Boolean] =
```
Naming follows the usual conventions: `spark.comet.legacy.` prefix,
`camelCase` final segment, and
-a `COMET_LEGACY_*` Scala symbol. The doc string must state which release
changed the behavior and
-what the old behavior was, so that `configs.md` is self-explanatory without
cross-referencing the
-upgrade guide.
+a `COMET_LEGACY_*` Scala symbol. No legacy config exists yet, so the first one
to land must also add
+the `CATEGORY_LEGACY` constant to `CometConf.scala` and a corresponding
section to `configs.md`. The
+doc string must state which release changed the behavior and what the old
behavior was, so that
+`configs.md` is self-explanatory without cross-referencing the upgrade guide.
The checklist for a behavior change:
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index 8ac9f1eea9..add59e103a 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -60,7 +60,7 @@ Comet provides the following configuration settings.
| `spark.comet.exec.columnarToRow.native.enabled` | Whether to enable native
columnar to row conversion. When enabled, Comet will use native Rust code to
convert Arrow columnar data to Spark UnsafeRow format instead of the JVM
implementation. The native conversion carries a fixed JNI cost per batch and is
slower than the JVM implementation for small batches. | false |
| `spark.comet.exec.enabled` | Whether to enable Comet native vectorized
execution for Spark. This controls whether Spark should convert operators into
their Comet counterparts and execute them in native space. Note: each operator
is associated with a separate config in the format of
`spark.comet.exec.<operator_name>.enabled` at the moment, and both the config
and this need to be turned on, in order for the operator to be executed in
native. | true |
| `spark.comet.exec.forceShuffledHashJoin` | Experimental feature to force
Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance.
This feature is not stable yet. For more information, refer to the [Comet
Tuning
Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). |
false |
-| `spark.comet.exec.pyarrowUdf.enabled` | Experimental: whether to enable
optimized execution of PyArrow UDFs (mapInArrow/mapInPandas). When enabled,
Comet passes Arrow columnar data directly to Python UDFs without the
intermediate Arrow-to-Row-to-Arrow conversion that Spark normally performs.
Disabled by default while the feature stabilizes. | false |
+| `spark.comet.exec.pyarrowUDF.enabled` | Experimental: whether to enable
optimized execution of PyArrow UDFs (mapInArrow/mapInPandas). When enabled,
Comet passes Arrow columnar data directly to Python UDFs without the
intermediate Arrow-to-Row-to-Arrow conversion that Spark normally performs.
Disabled by default while the feature stabilizes. | false |
| `spark.comet.exec.scalaUDF.codegen.enabled` | Whether to route Spark
`ScalaUDF` expressions through Comet's Arrow-direct codegen dispatcher. When
enabled, a supported ScalaUDF is compiled into a per-batch kernel that reads
and writes Arrow vectors directly from native execution. When disabled, plans
containing a ScalaUDF fall back to Spark for the enclosing operator. The same
dispatcher backs the regex family (`rlike`, `regexp_replace`, `split`,
`regexp_extract`, `regexp_extract_all`, [...]
| `spark.comet.exec.strictFloatingPoint` | When enabled, fall back to Spark
for floating-point operations that may differ from Spark, such as when
comparing or sorting -0.0 and 0.0. For more information, refer to the [Comet
Compatibility
Guide](https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html).
| false |
| `spark.comet.exec.transitionRevert.enabled` | When enabled, Comet reverts a
query stage to Spark row-based execution if the number of columnar-to-row (C2R)
transitions in the stage exceeds the configured threshold. This avoids the
overhead of repeated format conversions in stages where many operators fall
back to row-based execution. | false |
diff --git a/_sources/user-guide/latest/pyarrow-udfs.md.txt
b/_sources/user-guide/latest/pyarrow-udfs.md.txt
index 31cf932107..ff493de23c 100644
--- a/_sources/user-guide/latest/pyarrow-udfs.md.txt
+++ b/_sources/user-guide/latest/pyarrow-udfs.md.txt
@@ -77,14 +77,14 @@ CometMapInBatch <- Arrow batch in/out, Python
runner attached
The optimization is experimental and disabled by default. Enable it with:
```
-spark.comet.exec.pyarrowUdf.enabled=true
+spark.comet.exec.pyarrowUDF.enabled=true
```
The default is `false` while the feature stabilizes.
### Relationship to Spark's PySpark Arrow conversion conf
-`spark.comet.exec.pyarrowUdf.enabled` is **not** the same as PySpark's
+`spark.comet.exec.pyarrowUDF.enabled` is **not** the same as PySpark's
[`spark.sql.execution.arrow.pyspark.enabled`](https://spark.apache.org/docs/latest/api/python/tutorial/sql/arrow_pandas.html#enabling-for-conversion-to-from-pandas).
That conf controls whether Spark uses Arrow when materializing a DataFrame to
a Pandas DataFrame
(`toPandas()`) or constructing one from Pandas. The Comet conf controls a
planner rewrite for
@@ -110,7 +110,7 @@ spark = SparkSession.builder \
.config("spark.plugins", "org.apache.spark.CometPlugin") \
.config("spark.comet.enabled", "true") \
.config("spark.comet.exec.enabled", "true") \
- .config("spark.comet.exec.pyarrowUdf.enabled", "true") \
+ .config("spark.comet.exec.pyarrowUDF.enabled", "true") \
.config("spark.memory.offHeap.enabled", "true") \
.config("spark.memory.offHeap.size", "2g") \
.getOrCreate()
@@ -201,7 +201,7 @@ on the unoptimized path.
UDF that reads the Arrow field's time zone or localizes to wall-clock time
(for example a
`mapInPandas` UDF that strips the tz and treats the value as naive local
time): under a non-UTC
session time zone such a UDF can diverge from the unoptimized path. Set
- `spark.comet.exec.pyarrowUdf.enabled=false` for those UDFs.
+ `spark.comet.exec.pyarrowUDF.enabled=false` for those UDFs.
- `spark.sql.execution.arrow.useLargeVarTypes=true` is not supported. With
this conf enabled,
Spark widens `StringType` and `BinaryType` to Arrow's 8-byte-offset variants
in the
destination IPC root, while Comet's source vectors always use 4-byte
offsets. The buffer-copy
diff --git a/contributor-guide/config_conventions.html
b/contributor-guide/config_conventions.html
index 1dd61ace2b..1656503e53 100644
--- a/contributor-guide/config_conventions.html
+++ b/contributor-guide/config_conventions.html
@@ -560,9 +560,9 @@ broadest (the prefix) to narrowest (the specific
setting).</p>
prefix like <code class="docutils literal notranslate"><span
class="pre">spark.sql.comet.</span></code>.</p></li>
<li><p><strong>Category segment.</strong> The first segment after the prefix
names a broad area:
<code class="docutils literal notranslate"><span
class="pre">exec</span></code> (execution and expressions), <code
class="docutils literal notranslate"><span class="pre">scan</span></code>
(source readers), <code class="docutils literal notranslate"><span
class="pre">parquet</span></code> (Parquet-specific
-settings), <code class="docutils literal notranslate"><span
class="pre">shuffle</span></code> (shuffle behavior), <code class="docutils
literal notranslate"><span class="pre">columnar</span></code> (columnar
shuffle), <code class="docutils literal notranslate"><span
class="pre">explain</span></code> (plan
-explain/logging), <code class="docutils literal notranslate"><span
class="pre">metrics</span></code>, <code class="docutils literal
notranslate"><span class="pre">tracing</span></code>, <code class="docutils
literal notranslate"><span class="pre">testing</span></code>, <code
class="docutils literal notranslate"><span class="pre">convert</span></code>
(Spark → Comet source
-conversion), or <code class="docutils literal notranslate"><span
class="pre">expression</span></code> / <code class="docutils literal
notranslate"><span class="pre">operator</span></code> (per-expression /
per-operator flags).</p></li>
+settings), <code class="docutils literal notranslate"><span
class="pre">shuffle</span></code> (shuffle behavior), <code class="docutils
literal notranslate"><span class="pre">explain</span></code> (plan
explain/logging), <code class="docutils literal notranslate"><span
class="pre">metrics</span></code>,
+<code class="docutils literal notranslate"><span
class="pre">tracing</span></code>, <code class="docutils literal
notranslate"><span class="pre">debug</span></code>, <code class="docutils
literal notranslate"><span class="pre">testing</span></code>, <code
class="docutils literal notranslate"><span class="pre">convert</span></code>
(Spark → Comet source conversion), or
+<code class="docutils literal notranslate"><span
class="pre">expression</span></code> / <code class="docutils literal
notranslate"><span class="pre">operator</span></code> (per-expression /
per-operator flags).</p></li>
<li><p><strong>Segment casing.</strong> Multi-word segments use <code
class="docutils literal notranslate"><span class="pre">camelCase</span></code>,
not dot- or kebab-separated.
Prefer <code class="docutils literal notranslate"><span
class="pre">spark.comet.foo.maxThreadNum</span></code> over <code
class="docutils literal notranslate"><span
class="pre">spark.comet.foo.max.thread.num</span></code> or
<code class="docutils literal notranslate"><span
class="pre">spark.comet.foo.max-thread-num</span></code>.</p></li>
@@ -572,8 +572,8 @@ in <code class="docutils literal notranslate"><span
class="pre">.enabled</span><
a verb (<code class="docutils literal notranslate"><span
class="pre">force...</span></code>, <code class="docutils literal
notranslate"><span class="pre">allow...</span></code>) can omit <code
class="docutils literal notranslate"><span class="pre">.enabled</span></code>
because the verb already encodes
the action — for example <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.forceShuffledHashJoin</span></code>.</p></li>
<li><p><strong>Acronyms.</strong> Treat acronyms consistently as <code
class="docutils literal notranslate"><span class="pre">UDF</span></code>, <code
class="docutils literal notranslate"><span class="pre">SHJ</span></code>, <code
class="docutils literal notranslate"><span class="pre">SMJ</span></code>, <code
class="docutils literal notranslate"><span class="pre">IO</span></code> —
either all-caps
-or camelCase, but do not mix within one key (<code class="docutils literal
notranslate"><span class="pre">pyarrowUdf</span></code> and <code
class="docutils literal notranslate"><span class="pre">scalaUDF</span></code>
in the same
-cluster is a red flag).</p></li>
+or camelCase, but do not mix within one cluster (<code class="docutils literal
notranslate"><span class="pre">pyarrowUdf</span></code> alongside <code
class="docutils literal notranslate"><span class="pre">scalaUDF</span></code>
would
+be a red flag; both spell it <code class="docutils literal notranslate"><span
class="pre">UDF</span></code>).</p></li>
</ul>
</section>
<section id="symbol-naming-scala">
@@ -595,7 +595,7 @@ cluster is a red flag).</p></li>
<td><p><code class="docutils literal notranslate"><span
class="pre">COMET_FORCE_SHJ</span></code></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.parquet.rowFilterPushdown.enabled</span></code></p></td>
-<td><p><code class="docutils literal notranslate"><span
class="pre">COMET_ROW_FILTER_PUSHDOWN</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED</span></code></p></td>
</tr>
</tbody>
</table>
@@ -667,9 +667,10 @@ the previous behavior:</p>
</pre></div>
</div>
<p>Naming follows the usual conventions: <code class="docutils literal
notranslate"><span class="pre">spark.comet.legacy.</span></code> prefix, <code
class="docutils literal notranslate"><span class="pre">camelCase</span></code>
final segment, and
-a <code class="docutils literal notranslate"><span
class="pre">COMET_LEGACY_*</span></code> Scala symbol. The doc string must
state which release changed the behavior and
-what the old behavior was, so that <code class="docutils literal
notranslate"><span class="pre">configs.md</span></code> is self-explanatory
without cross-referencing the
-upgrade guide.</p>
+a <code class="docutils literal notranslate"><span
class="pre">COMET_LEGACY_*</span></code> Scala symbol. No legacy config exists
yet, so the first one to land must also add
+the <code class="docutils literal notranslate"><span
class="pre">CATEGORY_LEGACY</span></code> constant to <code class="docutils
literal notranslate"><span class="pre">CometConf.scala</span></code> and a
corresponding section to <code class="docutils literal notranslate"><span
class="pre">configs.md</span></code>. The
+doc string must state which release changed the behavior and what the old
behavior was, so that
+<code class="docutils literal notranslate"><span
class="pre">configs.md</span></code> is self-explanatory without
cross-referencing the upgrade guide.</p>
<p>The checklist for a behavior change:</p>
<ol class="arabic simple">
<li><p>Make the behavior change itself, gated on the new legacy
config.</p></li>
diff --git a/searchindex.js b/searchindex.js
index 45396097cb..15d1ea07d5 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[54, "id1"]], "%": [[52, "id1"]], "&":
[[42, "id1"]], "*": [[52, "id2"]], "+": [[52, "id3"]], "-": [[52, "id4"]], "/":
[[52, "id5"]], "1. Format Your Code": [[39, "format-your-code"]], "1. Install
Comet": [[60, "install-comet"], [70, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[27, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[39, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[60,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[54, "id1"]], "%": [[52, "id1"]], "&":
[[42, "id1"]], "*": [[52, "id2"]], "+": [[52, "id3"]], "-": [[52, "id4"]], "/":
[[52, "id5"]], "1. Format Your Code": [[39, "format-your-code"]], "1. Install
Comet": [[60, "install-comet"], [70, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[27, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[39, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[60,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index fbc0bc3df1..22df4fd641 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -851,7 +851,7 @@ under the License.
<td><p>Experimental feature to force Spark to replace SortMergeJoin with
ShuffledHashJoin for improved performance. This feature is not stable yet. For
more information, refer to the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/tuning.html";>Comet
Tuning Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.pyarrowUdf.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.pyarrowUDF.enabled</span></code></p></td>
<td><p>Experimental: whether to enable optimized execution of PyArrow UDFs
(mapInArrow/mapInPandas). When enabled, Comet passes Arrow columnar data
directly to Python UDFs without the intermediate Arrow-to-Row-to-Arrow
conversion that Spark normally performs. Disabled by default while the feature
stabilizes.</p></td>
<td><p>false</p></td>
</tr>
diff --git a/user-guide/latest/pyarrow-udfs.html
b/user-guide/latest/pyarrow-udfs.html
index 9375991fc2..8b9d329a7f 100644
--- a/user-guide/latest/pyarrow-udfs.html
+++ b/user-guide/latest/pyarrow-udfs.html
@@ -793,13 +793,13 @@ copies that remain.</p>
<section id="configuration">
<h2>Configuration<a class="headerlink" href="#configuration" title="Link to
this heading">#</a></h2>
<p>The optimization is experimental and disabled by default. Enable it
with:</p>
-<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">spark</span><span
class="o">.</span><span class="n">comet</span><span class="o">.</span><span
class="n">exec</span><span class="o">.</span><span
class="n">pyarrowUdf</span><span class="o">.</span><span
class="n">enabled</span><span class="o">=</span><span class="n">true</span>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">spark</span><span
class="o">.</span><span class="n">comet</span><span class="o">.</span><span
class="n">exec</span><span class="o">.</span><span
class="n">pyarrowUDF</span><span class="o">.</span><span
class="n">enabled</span><span class="o">=</span><span class="n">true</span>
</pre></div>
</div>
<p>The default is <code class="docutils literal notranslate"><span
class="pre">false</span></code> while the feature stabilizes.</p>
<section id="relationship-to-spark-s-pyspark-arrow-conversion-conf">
<h3>Relationship to Spark’s PySpark Arrow conversion conf<a class="headerlink"
href="#relationship-to-spark-s-pyspark-arrow-conversion-conf" title="Link to
this heading">#</a></h3>
-<p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.pyarrowUdf.enabled</span></code> is
<strong>not</strong> the same as PySpark’s
+<p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.pyarrowUDF.enabled</span></code> is
<strong>not</strong> the same as PySpark’s
<a class="reference external"
href="https://spark.apache.org/docs/latest/api/python/tutorial/sql/arrow_pandas.html#enabling-for-conversion-to-from-pandas";><code
class="docutils literal notranslate"><span
class="pre">spark.sql.execution.arrow.pyspark.enabled</span></code></a>.
That conf controls whether Spark uses Arrow when materializing a DataFrame to
a Pandas DataFrame
(<code class="docutils literal notranslate"><span
class="pre">toPandas()</span></code>) or constructing one from Pandas. The
Comet conf controls a planner rewrite for
@@ -846,7 +846,7 @@ worker. Both confs can be set independently.</p>
<span class="o">.</span><span class="n">config</span><span
class="p">(</span><span class="s2">"spark.plugins"</span><span
class="p">,</span> <span
class="s2">"org.apache.spark.CometPlugin"</span><span
class="p">)</span> \
<span class="o">.</span><span class="n">config</span><span
class="p">(</span><span class="s2">"spark.comet.enabled"</span><span
class="p">,</span> <span class="s2">"true"</span><span
class="p">)</span> \
<span class="o">.</span><span class="n">config</span><span
class="p">(</span><span
class="s2">"spark.comet.exec.enabled"</span><span class="p">,</span>
<span class="s2">"true"</span><span class="p">)</span> \
- <span class="o">.</span><span class="n">config</span><span
class="p">(</span><span
class="s2">"spark.comet.exec.pyarrowUdf.enabled"</span><span
class="p">,</span> <span class="s2">"true"</span><span
class="p">)</span> \
+ <span class="o">.</span><span class="n">config</span><span
class="p">(</span><span
class="s2">"spark.comet.exec.pyarrowUDF.enabled"</span><span
class="p">,</span> <span class="s2">"true"</span><span
class="p">)</span> \
<span class="o">.</span><span class="n">config</span><span
class="p">(</span><span
class="s2">"spark.memory.offHeap.enabled"</span><span
class="p">,</span> <span class="s2">"true"</span><span
class="p">)</span> \
<span class="o">.</span><span class="n">config</span><span
class="p">(</span><span
class="s2">"spark.memory.offHeap.size"</span><span class="p">,</span>
<span class="s2">"2g"</span><span class="p">)</span> \
<span class="o">.</span><span class="n">getOrCreate</span><span
class="p">()</span>
@@ -931,7 +931,7 @@ passthrough or value-based UDF round-trips identically. The
difference is only o
UDF that reads the Arrow field’s time zone or localizes to wall-clock time
(for example a
<code class="docutils literal notranslate"><span
class="pre">mapInPandas</span></code> UDF that strips the tz and treats the
value as naive local time): under a non-UTC
session time zone such a UDF can diverge from the unoptimized path. Set
-<code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.pyarrowUdf.enabled=false</span></code> for those
UDFs.</p></li>
+<code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.pyarrowUDF.enabled=false</span></code> for those
UDFs.</p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">spark.sql.execution.arrow.useLargeVarTypes=true</span></code> is
not supported. With this conf enabled,
Spark widens <code class="docutils literal notranslate"><span
class="pre">StringType</span></code> and <code class="docutils literal
notranslate"><span class="pre">BinaryType</span></code> to Arrow’s
8-byte-offset variants in the
destination IPC root, while Comet’s source vectors always use 4-byte offsets.
The buffer-copy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
