This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 735fa279fb Publish built docs triggered by
f46719b46b2442b4e82a24084eda9d4acad70ed5
735fa279fb is described below
commit 735fa279fbf54c718e872cca5610b733c2fb596f
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon Jul 20 14:49:19 2026 +0000
Publish built docs triggered by f46719b46b2442b4e82a24084eda9d4acad70ed5
---
.../expression-audits/datetime_funcs.md.txt | 8 ++++
.../contributor-guide/spark_configs_support.md.txt | 44 +++++++++++++--------
_sources/user-guide/latest/expressions.md.txt | 4 +-
.../expression-audits/datetime_funcs.html | 14 +++++++
contributor-guide/spark_configs_support.html | 46 +++++++++++++---------
searchindex.js | 2 +-
user-guide/latest/expressions.html | 8 ++--
7 files changed, 84 insertions(+), 42 deletions(-)
diff --git a/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt
b/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt
index b20fad5899..aad1945f5e 100644
--- a/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt
@@ -87,4 +87,12 @@
- Rewrites to `MakeTimestamp(failOnError = false)` and runs through the
codegen dispatcher (`CometMakeTimestamp`), so invalid inputs return NULL to
match Spark.
+## try_to_date
+
+- Spark 4.1+. Rewrites to `Cast(..., EvalMode.LEGACY)` (no format, native) or
`Cast(GetTimestamp(..., failOnError = false))` (with format, via the codegen
dispatcher) before Comet sees the plan. In non-ANSI mode the rewritten tree is
identical to `to_date`; invalid inputs return NULL to match Spark.
+
+## try_to_timestamp
+
+- Rewrites to `Cast(..., EvalMode.LEGACY)` (no format, native) or
`GetTimestamp(..., failOnError = false)` (with format, via the codegen
dispatcher) before Comet sees the plan. In non-ANSI mode the rewritten tree is
identical to `to_timestamp`; invalid inputs return NULL to match Spark.
+
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/spark_configs_support.md.txt
b/_sources/contributor-guide/spark_configs_support.md.txt
index fe1da3cd23..31c48adc83 100644
--- a/_sources/contributor-guide/spark_configs_support.md.txt
+++ b/_sources/contributor-guide/spark_configs_support.md.txt
@@ -39,10 +39,10 @@ The status column uses these values:
- `spark.sql.legacy.timeParserPolicy`
- Default: `EXCEPTION`
- - Status: Falls back (see notes)
- - Affected expressions: `date_format`, `from_unixtime`, `unix_timestamp`,
`to_unix_timestamp`, `to_timestamp`, `to_timestamp_ntz`, `to_date`,
`try_to_timestamp` (Spark 4+)
- - Spark versions checked: 3.4.3, 3.5.8, 4.0.1
- - Date: 2026-05-02
+ - Status: Partial (see notes)
+ - Affected expressions: `date_format`, `from_unixtime`, `unix_timestamp`,
`to_unix_timestamp`, `to_timestamp`, `to_timestamp_ntz`, `to_date`,
`try_to_date` (Spark 4.1+), `try_to_timestamp` (Spark 3.4+)
+ - Spark versions checked: 3.4.3, 3.5.8, 4.0.1, 4.1.2
+ - Date: 2026-07-18
## Audit Notes
@@ -76,27 +76,36 @@ across Spark 3.4, 3.5, 4.0, and 4.1. Three expression
classes mix in
`Cast` between strings and date / timestamp also reads the policy via the
default
formatters but is tested separately by `CometCastSuite` and is out of scope
here.
-**Comet status.** None of the listed expressions consult
`legacyTimeParserPolicy` in
-their Comet serde. The native implementations of `date_format`,
`from_unixtime`, and
-`unix_timestamp` use a fixed strftime-style mapping that does not vary with
policy;
-the remaining four (`to_unix_timestamp`, `to_timestamp`, `to_date`,
-`try_to_timestamp`) have no native implementation and fall back to Spark.
Today this
-works because:
+**Comet status.** None of the native implementations consult
+`legacyTimeParserPolicy` directly. Comet instead uses native implementations
only for
+policy-independent cases and routes policy-sensitive expressions through
Spark's own
+expression code via the Arrow-direct codegen dispatcher. In particular,
+`to_timestamp`, `to_timestamp_ntz`, `to_date`, `try_to_date`, and
+`try_to_timestamp` are rewritten by Spark to `Cast` or `GetTimestamp` before
Comet
+sees the plan. Supported casts run natively, while `GetTimestamp` runs through
the
+codegen dispatcher and therefore preserves Spark's selected parser policy
inside the
+Comet pipeline. Other affected expressions use a mixture of native,
codegen-dispatch,
+and fallback paths:
- `date_format` is `Compatible` only for a small whitelist of formats under
UTC; the
whitelisted formats happen to produce identical output under all three
policies.
-- `from_unixtime` is marked `Incompatible` and falls back unless
- `spark.comet.expression.FromUnixTime.allowIncompatible=true` is set.
+ Other formats route through the codegen dispatcher.
+- `from_unixtime` uses the native implementation for its default format when
+ `spark.comet.expression.FromUnixTime.allowIncompatible=true` is set;
otherwise
+ it routes through the codegen dispatcher.
- `unix_timestamp(<timestamp_or_date>)` does not call the formatter at all; the
string-input overload falls back.
+- `to_unix_timestamp` routes through the codegen dispatcher.
If a Comet contributor adds native string-format parsing or extends the
date_format
whitelist, this audit should be revisited and the policy must be honored
explicitly.
**Test coverage.** `spark/src/test/resources/sql-tests/expressions/datetime/`:
-- One ConfigMatrix file per expression covering convergent inputs under
- `LEGACY,CORRECTED,EXCEPTION` (`*_time_parser_policy.sql`).
+- One ConfigMatrix file per originally audited expression covering convergent
+ inputs under `LEGACY,CORRECTED,EXCEPTION` (`*_time_parser_policy.sql`).
+ `try_to_date`, added in Spark 4.1, has native-coverage tests in
+ `try_datetime.sql` but has not yet been added to the parser-policy matrix.
- Per-policy files locking in divergent behavior:
- `_legacy.sql` -- lenient inputs (single-digit fields, out-of-range values,
trailing characters) and legacy-only pattern tokens (`'aaaa'`).
@@ -107,6 +116,7 @@ whitelist, this audit should be revisited and the policy
must be honored explici
`INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER` at
parse time.
**Findings.** All 42 generated test cases pass on Spark 3.4.3, 3.5.8, and
4.0.1. No
-Comet bugs were uncovered by the audit. The tests use `query
spark_answer_only` so
-that result-correctness is enforced regardless of whether Comet runs the
expression
-natively or falls back.
+Comet bugs were uncovered by the original audit. The `try_to_timestamp` policy
tests
+use the default `query` mode to enforce both result correctness and execution
within
+the Comet pipeline. Tests for expressions with known fallback paths continue
to use
+`query spark_answer_only`.
diff --git a/_sources/user-guide/latest/expressions.md.txt
b/_sources/user-guide/latest/expressions.md.txt
index f45fa21444..ae49f27f87 100644
--- a/_sources/user-guide/latest/expressions.md.txt
+++ b/_sources/user-guide/latest/expressions.md.txt
@@ -294,9 +294,9 @@ The type-name conversion functions (`bigint`, `binary`,
`boolean`, `date`, `deci
| `trunc` | β
| |
| `try_make_interval` | π | Produces legacy CalendarInterval; tracked by
[#4540](https://github.com/apache/datafusion-comet/issues/4540) |
| `try_make_timestamp` | β
| |
-| `try_to_date` | π | Rewrites to `Cast`/`GetTimestamp` but currently falls
back; tracked by
[#4556](https://github.com/apache/datafusion-comet/issues/4556) |
+| `try_to_date` | β
| Rewrites to `Cast`/`GetTimestamp` before Comet sees the
plan; same support as `to_date` |
| `try_to_time` | π | Spark 4.1 TIME type; tracked by
[#4288](https://github.com/apache/datafusion-comet/issues/4288) |
-| `try_to_timestamp` | π | Rewrites to `Cast`/`GetTimestamp` but currently
falls back; tracked by
[#4556](https://github.com/apache/datafusion-comet/issues/4556) |
+| `try_to_timestamp` | β
| Rewrites to `Cast`/`GetTimestamp` before Comet sees
the plan; same support as `to_timestamp` |
| `unix_date` | β
| |
| `unix_micros` | β
| |
| `unix_millis` | β
| |
diff --git a/contributor-guide/expression-audits/datetime_funcs.html
b/contributor-guide/expression-audits/datetime_funcs.html
index b71b40d8ee..d973d581df 100644
--- a/contributor-guide/expression-audits/datetime_funcs.html
+++ b/contributor-guide/expression-audits/datetime_funcs.html
@@ -647,6 +647,18 @@ under the License.
<li><p>Rewrites to <code class="docutils literal notranslate"><span
class="pre">MakeTimestamp(failOnError</span> <span class="pre">=</span> <span
class="pre">false)</span></code> and runs through the codegen dispatcher (<code
class="docutils literal notranslate"><span
class="pre">CometMakeTimestamp</span></code>), so invalid inputs return NULL to
match Spark.</p></li>
</ul>
</section>
+<section id="try-to-date">
+<h2>try_to_date<a class="headerlink" href="#try-to-date" title="Link to this
heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 4.1+. Rewrites to <code class="docutils literal
notranslate"><span class="pre">Cast(...,</span> <span
class="pre">EvalMode.LEGACY)</span></code> (no format, native) or <code
class="docutils literal notranslate"><span
class="pre">Cast(GetTimestamp(...,</span> <span class="pre">failOnError</span>
<span class="pre">=</span> <span class="pre">false))</span></code> (with
format, via the codegen dispatcher) before Comet sees the plan. In non-ANSI
mode the rewritten tree is identic [...]
+</ul>
+</section>
+<section id="try-to-timestamp">
+<h2>try_to_timestamp<a class="headerlink" href="#try-to-timestamp" title="Link
to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Rewrites to <code class="docutils literal notranslate"><span
class="pre">Cast(...,</span> <span class="pre">EvalMode.LEGACY)</span></code>
(no format, native) or <code class="docutils literal notranslate"><span
class="pre">GetTimestamp(...,</span> <span class="pre">failOnError</span> <span
class="pre">=</span> <span class="pre">false)</span></code> (with format, via
the codegen dispatcher) before Comet sees the plan. In non-ANSI mode the
rewritten tree is identical to <code class= [...]
+</ul>
+</section>
</section>
@@ -711,6 +723,8 @@ under the License.
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#to-timestamp-ntz">to_timestamp_ntz</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#to-utc-timestamp">to_utc_timestamp</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#try-make-timestamp">try_make_timestamp</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#try-to-date">try_to_date</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#try-to-timestamp">try_to_timestamp</a></li>
</ul>
</nav></div>
diff --git a/contributor-guide/spark_configs_support.html
b/contributor-guide/spark_configs_support.html
index f453b8960c..eba31247de 100644
--- a/contributor-guide/spark_configs_support.html
+++ b/contributor-guide/spark_configs_support.html
@@ -567,10 +567,10 @@ config and always defers to Spark.</p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">spark.sql.legacy.timeParserPolicy</span></code></p>
<ul>
<li><p>Default: <code class="docutils literal notranslate"><span
class="pre">EXCEPTION</span></code></p></li>
-<li><p>Status: Falls back (see notes)</p></li>
-<li><p>Affected expressions: <code class="docutils literal notranslate"><span
class="pre">date_format</span></code>, <code class="docutils literal
notranslate"><span class="pre">from_unixtime</span></code>, <code
class="docutils literal notranslate"><span
class="pre">unix_timestamp</span></code>, <code class="docutils literal
notranslate"><span class="pre">to_unix_timestamp</span></code>, <code
class="docutils literal notranslate"><span
class="pre">to_timestamp</span></code>, <code class [...]
-<li><p>Spark versions checked: 3.4.3, 3.5.8, 4.0.1</p></li>
-<li><p>Date: 2026-05-02</p></li>
+<li><p>Status: Partial (see notes)</p></li>
+<li><p>Affected expressions: <code class="docutils literal notranslate"><span
class="pre">date_format</span></code>, <code class="docutils literal
notranslate"><span class="pre">from_unixtime</span></code>, <code
class="docutils literal notranslate"><span
class="pre">unix_timestamp</span></code>, <code class="docutils literal
notranslate"><span class="pre">to_unix_timestamp</span></code>, <code
class="docutils literal notranslate"><span
class="pre">to_timestamp</span></code>, <code class [...]
+<li><p>Spark versions checked: 3.4.3, 3.5.8, 4.0.1, 4.1.2</p></li>
+<li><p>Date: 2026-07-18</p></li>
</ul>
</li>
</ul>
@@ -606,26 +606,35 @@ across Spark 3.4, 3.5, 4.0, and 4.1. Three expression
classes mix in
</ul>
<p><code class="docutils literal notranslate"><span
class="pre">Cast</span></code> between strings and date / timestamp also reads
the policy via the default
formatters but is tested separately by <code class="docutils literal
notranslate"><span class="pre">CometCastSuite</span></code> and is out of scope
here.</p>
-<p><strong>Comet status.</strong> None of the listed expressions consult <code
class="docutils literal notranslate"><span
class="pre">legacyTimeParserPolicy</span></code> in
-their Comet serde. The native implementations of <code class="docutils literal
notranslate"><span class="pre">date_format</span></code>, <code class="docutils
literal notranslate"><span class="pre">from_unixtime</span></code>, and
-<code class="docutils literal notranslate"><span
class="pre">unix_timestamp</span></code> use a fixed strftime-style mapping
that does not vary with policy;
-the remaining four (<code class="docutils literal notranslate"><span
class="pre">to_unix_timestamp</span></code>, <code class="docutils literal
notranslate"><span class="pre">to_timestamp</span></code>, <code
class="docutils literal notranslate"><span class="pre">to_date</span></code>,
-<code class="docutils literal notranslate"><span
class="pre">try_to_timestamp</span></code>) have no native implementation and
fall back to Spark. Today this
-works because:</p>
+<p><strong>Comet status.</strong> None of the native implementations consult
+<code class="docutils literal notranslate"><span
class="pre">legacyTimeParserPolicy</span></code> directly. Comet instead uses
native implementations only for
+policy-independent cases and routes policy-sensitive expressions through
Sparkβs own
+expression code via the Arrow-direct codegen dispatcher. In particular,
+<code class="docutils literal notranslate"><span
class="pre">to_timestamp</span></code>, <code class="docutils literal
notranslate"><span class="pre">to_timestamp_ntz</span></code>, <code
class="docutils literal notranslate"><span class="pre">to_date</span></code>,
<code class="docutils literal notranslate"><span
class="pre">try_to_date</span></code>, and
+<code class="docutils literal notranslate"><span
class="pre">try_to_timestamp</span></code> are rewritten by Spark to <code
class="docutils literal notranslate"><span class="pre">Cast</span></code> or
<code class="docutils literal notranslate"><span
class="pre">GetTimestamp</span></code> before Comet
+sees the plan. Supported casts run natively, while <code class="docutils
literal notranslate"><span class="pre">GetTimestamp</span></code> runs through
the
+codegen dispatcher and therefore preserves Sparkβs selected parser policy
inside the
+Comet pipeline. Other affected expressions use a mixture of native,
codegen-dispatch,
+and fallback paths:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span
class="pre">date_format</span></code> is <code class="docutils literal
notranslate"><span class="pre">Compatible</span></code> only for a small
whitelist of formats under UTC; the
-whitelisted formats happen to produce identical output under all three
policies.</p></li>
-<li><p><code class="docutils literal notranslate"><span
class="pre">from_unixtime</span></code> is marked <code class="docutils literal
notranslate"><span class="pre">Incompatible</span></code> and falls back unless
-<code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.FromUnixTime.allowIncompatible=true</span></code>
is set.</p></li>
+whitelisted formats happen to produce identical output under all three
policies.
+Other formats route through the codegen dispatcher.</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">from_unixtime</span></code> uses the native implementation for its
default format when
+<code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.FromUnixTime.allowIncompatible=true</span></code>
is set; otherwise
+it routes through the codegen dispatcher.</p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">unix_timestamp(<timestamp_or_date>)</span></code> does not
call the formatter at all; the
string-input overload falls back.</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">to_unix_timestamp</span></code> routes through the codegen
dispatcher.</p></li>
</ul>
<p>If a Comet contributor adds native string-format parsing or extends the
date_format
whitelist, this audit should be revisited and the policy must be honored
explicitly.</p>
<p><strong>Test coverage.</strong> <code class="docutils literal
notranslate"><span
class="pre">spark/src/test/resources/sql-tests/expressions/datetime/</span></code>:</p>
<ul class="simple">
-<li><p>One ConfigMatrix file per expression covering convergent inputs under
-<code class="docutils literal notranslate"><span
class="pre">LEGACY,CORRECTED,EXCEPTION</span></code> (<code class="docutils
literal notranslate"><span
class="pre">*_time_parser_policy.sql</span></code>).</p></li>
+<li><p>One ConfigMatrix file per originally audited expression covering
convergent
+inputs under <code class="docutils literal notranslate"><span
class="pre">LEGACY,CORRECTED,EXCEPTION</span></code> (<code class="docutils
literal notranslate"><span class="pre">*_time_parser_policy.sql</span></code>).
+<code class="docutils literal notranslate"><span
class="pre">try_to_date</span></code>, added in Spark 4.1, has native-coverage
tests in
+<code class="docutils literal notranslate"><span
class="pre">try_datetime.sql</span></code> but has not yet been added to the
parser-policy matrix.</p></li>
<li><p>Per-policy files locking in divergent behavior:</p>
<ul>
<li><p><code class="docutils literal notranslate"><span
class="pre">_legacy.sql</span></code> β lenient inputs (single-digit fields,
out-of-range values,
@@ -639,9 +648,10 @@ creation.</p></li>
</li>
</ul>
<p><strong>Findings.</strong> All 42 generated test cases pass on Spark 3.4.3,
3.5.8, and 4.0.1. No
-Comet bugs were uncovered by the audit. The tests use <code class="docutils
literal notranslate"><span class="pre">query</span> <span
class="pre">spark_answer_only</span></code> so
-that result-correctness is enforced regardless of whether Comet runs the
expression
-natively or falls back.</p>
+Comet bugs were uncovered by the original audit. The <code class="docutils
literal notranslate"><span class="pre">try_to_timestamp</span></code> policy
tests
+use the default <code class="docutils literal notranslate"><span
class="pre">query</span></code> mode to enforce both result correctness and
execution within
+the Comet pipeline. Tests for expressions with known fallback paths continue
to use
+<code class="docutils literal notranslate"><span class="pre">query</span>
<span class="pre">spark_answer_only</span></code>.</p>
</section>
</section>
</section>
diff --git a/searchindex.js b/searchindex.js
index c04f3f60e9..2085ca8f31 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[52, "id1"]], "%": [[50, "id1"]], "&":
[[40, "id1"]], "*": [[50, "id2"]], "+": [[50, "id3"]], "-": [[50, "id4"]], "/":
[[50, "id5"]], "1. Format Your Code": [[37, "format-your-code"]], "1. Install
Comet": [[58, "install-comet"], [68, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[26, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[37, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[58,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[52, "id1"]], "%": [[50, "id1"]], "&":
[[40, "id1"]], "*": [[50, "id2"]], "+": [[50, "id3"]], "-": [[50, "id4"]], "/":
[[50, "id5"]], "1. Format Your Code": [[37, "format-your-code"]], "1. Install
Comet": [[58, "install-comet"], [68, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[26, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[37, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[58,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/expressions.html
b/user-guide/latest/expressions.html
index 6665a8a49c..df46cc0078 100644
--- a/user-guide/latest/expressions.html
+++ b/user-guide/latest/expressions.html
@@ -1609,16 +1609,16 @@ expressions. The following function families are
<strong>not currently planned</
<td><p></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">try_to_date</span></code></p></td>
-<td><p>π</p></td>
-<td><p>Rewrites to <code class="docutils literal notranslate"><span
class="pre">Cast</span></code>/<code class="docutils literal notranslate"><span
class="pre">GetTimestamp</span></code> but currently falls back; tracked by <a
class="reference external"
href="https://github.com/apache/datafusion-comet/issues/4556";>#4556</a></p></td>
+<td><p>β
</p></td>
+<td><p>Rewrites to <code class="docutils literal notranslate"><span
class="pre">Cast</span></code>/<code class="docutils literal notranslate"><span
class="pre">GetTimestamp</span></code> before Comet sees the plan; same support
as <code class="docutils literal notranslate"><span
class="pre">to_date</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">try_to_time</span></code></p></td>
<td><p>π</p></td>
<td><p>Spark 4.1 TIME type; tracked by <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/4288";>#4288</a></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">try_to_timestamp</span></code></p></td>
-<td><p>π</p></td>
-<td><p>Rewrites to <code class="docutils literal notranslate"><span
class="pre">Cast</span></code>/<code class="docutils literal notranslate"><span
class="pre">GetTimestamp</span></code> but currently falls back; tracked by <a
class="reference external"
href="https://github.com/apache/datafusion-comet/issues/4556";>#4556</a></p></td>
+<td><p>β
</p></td>
+<td><p>Rewrites to <code class="docutils literal notranslate"><span
class="pre">Cast</span></code>/<code class="docutils literal notranslate"><span
class="pre">GetTimestamp</span></code> before Comet sees the plan; same support
as <code class="docutils literal notranslate"><span
class="pre">to_timestamp</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">unix_date</span></code></p></td>
<td><p>β
</p></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
