This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new a2a11dd802ee [SPARK-57030][SPARK-57031][DOC] Document the TIME data
type, functions, and ANSI behavior in the SQL reference
a2a11dd802ee is described below
commit a2a11dd802eecd1644c0de6a683bd726f8060493
Author: Maxim Gekk <[email protected]>
AuthorDate: Fri Jun 26 09:07:50 2026 +0200
[SPARK-57030][SPARK-57031][DOC] Document the TIME data type, functions, and
ANSI behavior in the SQL reference
### What changes were proposed in this pull request?
This PR completes the user-facing SQL reference documentation for the
`TIME` data type:
- `docs/sql-ref-datatypes.md`: add the missing `TimeType` rows to the
Python (`datetime.time` / `TimeType()`), R (`Not supported`), and SQL type-name
(`TIME, TIME(p)`) tables, and update the `TimeType(precision)` description to
reflect the supported precision range (`0` to `9`, default `6`).
- `docs/sql-ref-literals.md`: extend the `TIME` literal syntax to allow up
to 9 fractional-second digits and add a nanosecond-precision example.
- `docs/sql-ref-ansi-compliance.md`: document that `TIME` does not promote
to other types, the least common type of `TIME(n)`/`TIME(m)` is `TIME(max(n,
m))`, and Spark's deviations from the SQL standard (default precision `6` vs
ANSI `0`; `TIME WITH TIME ZONE` not supported).
The TIME-related functions and operators (`current_time`, `make_time`,
`to_time`, `try_to_time`, `time_trunc`, `time_diff`, `time_from_*`,
`time_to_*`, `hour`/`minute`/`second`) are already covered by the
auto-generated SQL function reference, which is built from the
`ExpressionDescription` annotations on the corresponding expressions and
registered in `FunctionRegistry`.
This PR addresses both
[SPARK-57030](https://issues.apache.org/jira/browse/SPARK-57030) (data-type
reference page) and
[SPARK-57031](https://issues.apache.org/jira/browse/SPARK-57031)
(functions/operators and ANSI compliance page).
### Why are the changes needed?
To finish documenting the `TIME` data type and its functions/operators and
ANSI behavior in the SQL reference.
### Does this PR introduce _any_ user-facing change?
No. Documentation-only changes.
### How was this patch tested?
Reviewed the rendered Markdown tables and verified the claims against the
implementation (`TimeType`, `DataTypeAstBuilder`, the `TIME` literal parser in
`AstBuilder`, and `FunctionRegistry`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56771 from MaxGekk/time-docs.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit 24434c3bf7b964e8d750c9632705554ee2421f7a)
Signed-off-by: Max Gekk <[email protected]>
---
docs/sql-data-sources-avro.md | 10 ++++++++++
docs/sql-data-sources-csv.md | 6 ++++++
docs/sql-data-sources-json.md | 8 +++++++-
docs/sql-data-sources-xml.md | 7 +++++++
docs/sql-ref-ansi-compliance.md | 2 ++
docs/sql-ref-datatypes.md | 5 ++++-
docs/sql-ref-datetime-pattern.md | 6 +++---
docs/sql-ref-functions-builtin.md | 2 +-
docs/sql-ref-functions.md | 4 ++--
docs/sql-ref-literals.md | 8 +++++++-
docs/sql-ref-syntax-aux-describe-table.md | 1 +
.../spark/sql/catalyst/expressions/datetimeExpressions.scala | 10 +++++++---
12 files changed, 57 insertions(+), 12 deletions(-)
diff --git a/docs/sql-data-sources-avro.md b/docs/sql-data-sources-avro.md
index 0c0d2dcbe243..a55cb00f9bd0 100644
--- a/docs/sql-data-sources-avro.md
+++ b/docs/sql-data-sources-avro.md
@@ -560,6 +560,11 @@ It also supports reading the following Avro [logical
types](https://avro.apache.
<td>long</td>
<td>TimestampType</td>
</tr>
+ <tr>
+ <td>time-micros</td>
+ <td>long</td>
+ <td>TimeType</td>
+ </tr>
<tr>
<td>decimal</td>
<td>fixed</td>
@@ -603,6 +608,11 @@ Spark supports writing of all Spark SQL types into Avro.
For most types, the map
<td>long</td>
<td>timestamp-micros</td>
</tr>
+ <tr>
+ <td>TimeType</td>
+ <td>long</td>
+ <td>time-micros</td>
+ </tr>
<tr>
<td>DecimalType</td>
<td>fixed</td>
diff --git a/docs/sql-data-sources-csv.md b/docs/sql-data-sources-csv.md
index a88b4ab14986..9dfe9739b7ac 100644
--- a/docs/sql-data-sources-csv.md
+++ b/docs/sql-data-sources-csv.md
@@ -180,6 +180,12 @@ Data source options of CSV can be set via:
<td>Sets the string that indicates a timestamp without timezone format.
Custom date formats follow the formats at <a
href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html";>Datetime
Patterns</a>. This applies to timestamp without timezone type, note that
zone-offset and time-zone components are not supported when writing or reading
this data type.</td>
<td>read/write</td>
</tr>
+ <tr>
+ <td><code>timeFormat</code></td>
+ <td>HH:mm:ss</td>
+ <td>Sets the string that indicates a time format. Custom time formats
follow the formats at <a
href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html";>Datetime
Patterns</a>. This applies to time type.</td>
+ <td>read/write</td>
+ </tr>
<tr>
<td><code>enableDateTimeParsingFallback</code></td>
<td>Enabled if the time parser policy has legacy settings or if no custom
date or timestamp pattern was provided.</td>
diff --git a/docs/sql-data-sources-json.md b/docs/sql-data-sources-json.md
index e9449ee17a4d..19e724deb0bf 100644
--- a/docs/sql-data-sources-json.md
+++ b/docs/sql-data-sources-json.md
@@ -201,10 +201,16 @@ Data source options of JSON can be set via:
<td>Sets the string that indicates a timestamp without timezone format.
Custom date formats follow the formats at <a
href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html";>Datetime
Patterns</a>. This applies to timestamp without timezone type, note that
zone-offset and time-zone components are not supported when writing or reading
this data type.</td>
<td>read/write</td>
</tr>
+ <tr>
+ <td><code>timeFormat</code></td>
+ <td>HH:mm:ss</td>
+ <td>Sets the string that indicates a time format. Custom time formats
follow the formats at <a
href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html";>Datetime
Patterns</a>. This applies to time type.</td>
+ <td>read/write</td>
+ </tr>
<tr>
<td><code>inferTimestamp</code></td>
<td><code>false</code></td>
- <td>Allows inferring of <code>TimestampType</code> and
<code>TimestampNTZType</code> from strings that match the timestamp patterns
defined by the <code>timestampFormat</code> and <code>timestampNTZFormat</code>
options respectively.</td>
+ <td>Allows inferring of <code>TimestampType</code> and
<code>TimestampNTZType</code> from strings that match the timestamp patterns
defined by the <code>timestampFormat</code> and <code>timestampNTZFormat</code>
options respectively. When enabled, it also allows inferring of
<code>TimeType</code> from strings that match the time pattern defined by the
<code>timeFormat</code> option.</td>
<td>read</td>
</tr>
<tr>
diff --git a/docs/sql-data-sources-xml.md b/docs/sql-data-sources-xml.md
index bdc0c7e066c3..e714ded0ee28 100644
--- a/docs/sql-data-sources-xml.md
+++ b/docs/sql-data-sources-xml.md
@@ -188,6 +188,13 @@ Data source options of XML can be set via:
<td>read/write</td>
</tr>
+ <tr>
+ <td><code>timeFormat</code></td>
+ <td><code>HH:mm:ss</code></td>
+ <td>Sets the string that indicates a time format. Custom time formats
follow the formats at <a
href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html";>
datetime pattern</a>. This applies to time type.</td>
+ <td>read/write</td>
+ </tr>
+
<tr>
<td><code>locale</code></td>
<td><code>en-US</code></td>
diff --git a/docs/sql-ref-ansi-compliance.md b/docs/sql-ref-ansi-compliance.md
index bd449524895c..ea89e8cea349 100644
--- a/docs/sql-ref-ansi-compliance.md
+++ b/docs/sql-ref-ansi-compliance.md
@@ -257,6 +257,8 @@ At the heart of this conflict resolution is the Type
Precedence List which defin
\*\*\* For a complex type, the precedence rule applies recursively to its
component elements.
+The `TIME` type does not promote to any other type. Note that Spark's `TIME`
type deviates from the SQL standard in two ways: the default fractional-seconds
precision is `6` (the ANSI default is `0`), and `TIME WITH TIME ZONE` is not
supported.
+
Special rules apply for untyped NULL. A NULL can be promoted to any other type.
This is a graphical depiction of the precedence list as a directed tree:
diff --git a/docs/sql-ref-datatypes.md b/docs/sql-ref-datatypes.md
index e0d36631dec0..27663763d6bd 100644
--- a/docs/sql-ref-datatypes.md
+++ b/docs/sql-ref-datatypes.md
@@ -47,7 +47,7 @@ Spark SQL and DataFrames support the following data types:
- `DateType`: Represents values comprising values of fields year, month and
day, without a
time-zone.
- `TimeType(precision)`: Represents values comprising values of fields hour,
minute and second with the number of decimal digits `precision` following the
decimal point in the seconds field, without a time-zone.
- The range of values is from `00:00:00` to `23:59:59` for min precision `0`,
and to `23:59:59.999999` for max precision `6`.
+ The range of values is from `00:00:00` to `23:59:59` for min precision `0`,
and to `23:59:59.999999999` for max precision `9`. The default precision is `6`.
- `TimestampType`: Timestamp with local time zone(TIMESTAMP_LTZ). It
represents values comprising values of fields year, month, day,
hour, minute, and second, with the session local time-zone. The timestamp
value represents an
absolute point in time.
@@ -143,6 +143,7 @@ from pyspark.sql.types import *
|**TimestampType**|datetime.datetime|TimestampType()|
|**TimestampNTZType**|datetime.datetime|TimestampNTZType()|
|**DateType**|datetime.date|DateType()|
+|**TimeType**|datetime.time|TimeType()|
|**DayTimeIntervalType**|datetime.timedelta|DayTimeIntervalType()|
|**GeometryType**|Geometry|GeometryType(*srid*)<br/>**Note:** *srid* is
required and may be an `int` or the string `"ANY"`.|
|**GeographyType**|Geography|GeographyType(*srid*)<br/>**Note:** *srid* is
required and may be an `int` or the string `"ANY"`.|
@@ -241,6 +242,7 @@ please use factory methods provided in
|**BooleanType**|logical|"bool"|
|**TimestampType**|POSIXct|"timestamp"|
|**DateType**|Date|"date"|
+|**TimeType**|Not supported|Not supported|
|**GeometryType**|Not supported|Not supported|
|**GeographyType**|Not supported|Not supported|
|**ArrayType**|vector or list|list(type="array", elementType=*elementType*,
containsNull=[*containsNull*])<br/>**Note:** The default value of
*containsNull* is TRUE.|
@@ -264,6 +266,7 @@ The following table shows the type names as well as aliases
used in Spark SQL pa
|**FloatType**|FLOAT, REAL|
|**DoubleType**|DOUBLE|
|**DateType**|DATE|
+|**TimeType**|TIME, TIME(p)|
|**TimestampType**|TIMESTAMP, TIMESTAMP_LTZ|
|**TimestampNTZType**|TIMESTAMP_NTZ|
|**StringType**|STRING|
diff --git a/docs/sql-ref-datetime-pattern.md b/docs/sql-ref-datetime-pattern.md
index e5d5388f262e..9ce91738bb49 100644
--- a/docs/sql-ref-datetime-pattern.md
+++ b/docs/sql-ref-datetime-pattern.md
@@ -23,10 +23,10 @@ There are several common scenarios for datetime usage in
Spark:
- CSV/JSON datasources use the pattern string for parsing and formatting
datetime content.
-- Datetime functions related to convert `StringType` to/from `DateType` or
`TimestampType`.
- For example, `unix_timestamp`, `date_format`, `to_unix_timestamp`,
`from_unixtime`, `to_date`, `to_timestamp`, `from_utc_timestamp`,
`to_utc_timestamp`, etc.
+- Datetime functions related to convert `StringType` to/from `DateType`,
`TimeType` or `TimestampType`.
+ For example, `unix_timestamp`, `date_format`, `to_unix_timestamp`,
`from_unixtime`, `to_date`, `to_time`, `to_timestamp`, `from_utc_timestamp`,
`to_utc_timestamp`, etc.
-Spark uses pattern letters in the following table for date and timestamp
parsing and formatting:
+Spark uses pattern letters in the following table for date, time and timestamp
parsing and formatting:
|Symbol|Meaning|Presentation|Examples|
|------|-------|------------|--------|
diff --git a/docs/sql-ref-functions-builtin.md
b/docs/sql-ref-functions-builtin.md
index 22e52d0500c5..0a8cce0e7b4d 100644
--- a/docs/sql-ref-functions-builtin.md
+++ b/docs/sql-ref-functions-builtin.md
@@ -51,7 +51,7 @@ user-defined function that may share the same name.
#### Examples
{% include_api_gen generated-map-funcs-examples.html %}
-### Date and Timestamp Functions
+### Date, Time and Timestamp Functions
{% include_api_gen generated-datetime-funcs-table.html %}
#### Examples
{% include_api_gen generated-datetime-funcs-examples.html %}
diff --git a/docs/sql-ref-functions.md b/docs/sql-ref-functions.md
index c218c2a65981..2b50dbe862e0 100644
--- a/docs/sql-ref-functions.md
+++ b/docs/sql-ref-functions.md
@@ -24,7 +24,7 @@ Built-in functions are commonly used routines that Spark SQL
predefines and a co
### Built-in Functions
-Spark SQL has some categories of frequently-used built-in functions for
aggregation, arrays/maps, date/timestamp, and JSON data.
+Spark SQL has some categories of frequently-used built-in functions for
aggregation, arrays/maps, date/time/timestamp, and JSON data.
This subsection presents the usages and descriptions of these functions.
#### Scalar Functions
@@ -32,7 +32,7 @@ This subsection presents the usages and descriptions of these
functions.
* [Collection Functions](sql-ref-functions-builtin.html#collection-functions)
* [Struct Functions](sql-ref-functions-builtin.html#struct-functions)
* [Map Functions](sql-ref-functions-builtin.html#map-functions)
- * [Date and Timestamp
Functions](sql-ref-functions-builtin.html#date-and-timestamp-functions)
+ * [Date, Time and Timestamp
Functions](sql-ref-functions-builtin.html#date-time-and-timestamp-functions)
* [Mathematical
Functions](sql-ref-functions-builtin.html#mathematical-functions)
* [String Functions](sql-ref-functions-builtin.html#string-functions)
* [Bitwise Functions](sql-ref-functions-builtin.html#bitwise-functions)
diff --git a/docs/sql-ref-literals.md b/docs/sql-ref-literals.md
index 5f4f8930581e..b016c713dddf 100644
--- a/docs/sql-ref-literals.md
+++ b/docs/sql-ref-literals.md
@@ -433,7 +433,7 @@ SELECT DATE '2011-11-11' AS col;
```sql
TIME { '[h]h:[m]m[:]' |
'[h]h:[m]m:[s]s[.]' |
- '[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us]'}
+ '[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][ns][ns][ns]'}
```
**Note:** defaults to `00` if hour, minute or second is not specified.
@@ -464,6 +464,12 @@ SELECT TIME'23:59:59.999999' as col;
+---------------+
|23:59:59.999999|
+---------------+
+SELECT TIME'23:59:59.999999999' as col;
++------------------+
+|col |
++------------------+
+|23:59:59.999999999|
++------------------+
```
#### Timestamp Syntax
diff --git a/docs/sql-ref-syntax-aux-describe-table.md
b/docs/sql-ref-syntax-aux-describe-table.md
index cb84b0c7fefb..5c417689dde2 100644
--- a/docs/sql-ref-syntax-aux-describe-table.md
+++ b/docs/sql-ref-syntax-aux-describe-table.md
@@ -154,6 +154,7 @@ to return the metadata pertaining to a partition or column
respectively.
| BinaryType | `{ "name" : "binary" }`
|
| BooleanType | `{ "name" : "boolean" }`
|
| DateType | `{ "name" : "date" }`
|
+| TimeType | `{ "name" : "time(p)" }`
|
| VariantType | `{ "name" : "variant" }`
|
| TimestampType | `{ "name" : "timestamp_ltz" }`
|
| TimestampNTZType | `{ "name" : "timestamp_ntz" }`
|
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
index 667f79c37288..a2ed109ec66c 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
@@ -1209,7 +1209,7 @@ case class DayName(child: Expression) extends
GetDateField with DefaultStringPro
usage = "_FUNC_(timestamp, fmt) - Converts `timestamp` to a value of string
in the format specified by the date format `fmt`.",
arguments = """
Arguments:
- * timestamp - A date/timestamp or string to be converted to the given
format.
+ * timestamp - A date, time, timestamp or string to be converted to the
given format.
* fmt - Date/time format pattern to follow. See <a
href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html";>Datetime
Patterns</a> for valid date
and time format patterns.
""",
@@ -1217,6 +1217,8 @@ case class DayName(child: Expression) extends
GetDateField with DefaultStringPro
Examples:
> SELECT _FUNC_('2016-04-08', 'y');
2016
+ > SELECT _FUNC_(TIME'14:30:45', 'HH:mm:ss');
+ 14:30:45
""",
group = "datetime_funcs",
since = "1.5.0")
@@ -3874,11 +3876,11 @@ object DatePart {
// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "_FUNC_(field, source) - Extracts a part of the date/timestamp or
interval source.",
+ usage = "_FUNC_(field, source) - Extracts a part of the date, time,
timestamp, or interval source.",
arguments = """
Arguments:
* field - selects which part of the source should be extracted, and
supported string values are as same as the fields of the equivalent function
`EXTRACT`.
- * source - a date/timestamp or interval column from where `field` should
be extracted
+ * source - a date, time, timestamp, or interval column from where
`field` should be extracted
""",
examples = """
Examples:
@@ -3898,6 +3900,8 @@ object DatePart {
11
> SELECT _FUNC_('MINUTE', INTERVAL '123 23:55:59.002001' DAY TO SECOND);
55
+ > SELECT _FUNC_('HOUR', TIME'09:08:01.000001');
+ 9
""",
note = """
The _FUNC_ function is equivalent to the SQL-standard function
`EXTRACT(field FROM source)`
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
