This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 42a48bbdf9 chore: remove unused spark.comet.exceptionOnDatetimeRebase
config (#5221)
42a48bbdf9 is described below
commit 42a48bbdf9e14735a1bebeeb6545ab7979e6ca9b
Author: Andy Grove <[email protected]>
AuthorDate: Sun Aug 2 14:15:03 2026 -0600
chore: remove unused spark.comet.exceptionOnDatetimeRebase config (#5221)
This config was never read anywhere in the codebase. Setting it had no
effect: Comet never raised on dates/timestamps written with the legacy
hybrid Julian/Gregorian calendar. Remove the dead config and drop the
note about it from the Parquet scan compatibility guide, which already
documents the underlying lack of datetime rebasing support (#5010).
---
docs/source/user-guide/latest/compatibility/scans.md | 4 +---
spark/src/main/scala/org/apache/comet/CometConf.scala | 12 ------------
2 files changed, 1 insertion(+), 15 deletions(-)
diff --git a/docs/source/user-guide/latest/compatibility/scans.md
b/docs/source/user-guide/latest/compatibility/scans.md
index 0c07b5c237..00b3284e10 100644
--- a/docs/source/user-guide/latest/compatibility/scans.md
+++ b/docs/source/user-guide/latest/compatibility/scans.md
@@ -57,9 +57,7 @@ The following limitation may produce incorrect results
without falling back to S
if they were written using the Proleptic Gregorian calendar. This produces
silently-wrong
values for dates before October 15, 1582 in both projections and predicates.
Comet also
ignores `spark.sql.parquet.datetimeRebaseModeInRead` and the file-level
- `org.apache.spark.legacyDateTime` metadata that would tell it to rebase. The
- `spark.comet.exceptionOnDatetimeRebase` config is currently dead code and
does not raise on
- legacy-calendar data. Tracked by
+ `org.apache.spark.legacyDateTime` metadata that would tell it to rebase.
Tracked by
[#5010](https://github.com/apache/datafusion-comet/issues/5010).
The following limitations raise an error at scan time rather than falling back
to Spark:
diff --git a/spark/src/main/scala/org/apache/comet/CometConf.scala
b/spark/src/main/scala/org/apache/comet/CometConf.scala
index cdbc430829..ae29715193 100644
--- a/spark/src/main/scala/org/apache/comet/CometConf.scala
+++ b/spark/src/main/scala/org/apache/comet/CometConf.scala
@@ -715,18 +715,6 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(false)
- val COMET_EXCEPTION_ON_LEGACY_DATE_TIMESTAMP: ConfigEntry[Boolean] =
- conf("spark.comet.exceptionOnDatetimeRebase")
- .category(CATEGORY_EXEC)
- .doc("Whether to throw exception when seeing dates/timestamps from the
legacy hybrid " +
- "(Julian + Gregorian) calendar. Since Spark 3, dates/timestamps were
written according " +
- "to the Proleptic Gregorian calendar. When this is true, Comet will " +
- "throw exceptions when seeing these dates/timestamps that were written
by Spark version " +
- "before 3.0. If this is false, these dates/timestamps will be read as
if they were " +
- "written to the Proleptic Gregorian calendar and will not be rebased.")
- .booleanConf
- .createWithDefault(false)
-
val COMET_ENABLE_PARTIAL_HASH_AGGREGATE: ConfigEntry[Boolean] =
conf("spark.comet.testing.aggregate.partialMode.enabled")
.internal()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
