mbutrovich commented on code in PR #329:
URL: https://github.com/apache/datafusion-spark/pull/329#discussion_r4615365652
##########
crates/spark/src/function/datetime/make_date.rs:
##########
@@ -0,0 +1,538 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+use std::sync::Arc;
+
+use arrow::array::{Array, Date32Array, Date32Builder, new_empty_array};
+use arrow::datatypes::{DataType, Field, FieldRef};
+use datafusion_common::cast::as_int32_array;
+use datafusion_common::types::{NativeType, logical_int32, logical_string};
+use datafusion_common::utils::take_function_args;
+use datafusion_common::{Result, ScalarValue, exec_err, internal_err};
+use datafusion_expr::{
+ Coercion, ColumnarValue, Documentation, ReturnFieldArgs,
ScalarFunctionArgs,
+ ScalarUDFImpl, Signature, TypeSignatureClass, Volatility,
+};
+use datafusion_macros::user_doc;
+
+const ARITHMETIC_OVERFLOW_ERROR: &str = r#"[ARITHMETIC_OVERFLOW] overflow. If
necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
SQLSTATE: 22003"#;
+
+/// Spark-compatible `make_date` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#make_date>
+#[user_doc(
+ doc_section(label = "Time and Date Functions"),
+ description = "Creates a date from `year`, `month` and `day` fields.
DataFusion has no built-in \
+ equivalent. Years from -999999999 to 999999999 are accepted (year 0 is
1 BC), as long as \
+ the date fits the DATE range. When the fields do not form a valid date
(for example \
+ month 13 or February 29 in a non-leap year), the function returns
NULL, or raises \
+ DATETIME_FIELD_OUT_OF_BOUNDS when
`datafusion.execution.enable_ansi_mode` is true. A \
+ valid date outside the DATE range (before -5877641-06-23 or after
+5881580-07-11) raises \
+ ARITHMETIC_OVERFLOW in both modes, as in Spark. Other numeric and
string inputs are cast \
+ to INT first, which truncates fractions and raises an error if the
value is out of the \
+ INT range or is not a valid number. These errors match Spark with ANSI
mode enabled. \
+ With ANSI mode disabled, Spark wraps an out-of-range BIGINT and
returns NULL for a \
+ malformed string, while this function still errors (see \
Review Comment:
> In 98e1f48 the doc sentence now uses your wording.
My wording missed one more case, sorry. With ANSI mode off, Spark's cast
from STRING to INT drops a fractional part, so `make_date('2024.7', '1', '1')`
and `make_date('2024', '1.9', '1')` both return `2024-01-01` on Spark 4.2.0.
Here both raise `Cannot cast string '2024.7' to value of Int32 type` (or
`'1.9'`) in both modes. With ANSI mode on, Spark raises `CAST_INVALID_INPUT`,
so that half already matches.
Could the sentence ([`make_date.rs`
L35-L37](https://github.com/apache/datafusion-spark/blob/98e1f480e5444c9ddf550bf8644e073fe2fb74d2/crates/spark/src/function/datetime/make_date.rs#L35-L37))
read like this?
```
With ANSI mode disabled, Spark wraps an out-of-range BIGINT or DECIMAL,
truncates a string
with a fractional part, returns NULL for a malformed string or a
floating-point value out of
the INT range, and casts NaN to 0, while this function still errors
```
Could `make_date.slt` also record it next to the malformed-string query
([L110-L115](https://github.com/apache/datafusion-spark/blob/98e1f480e5444c9ddf550bf8644e073fe2fb74d2/crates/sqllogictest/test_files/datetime/make_date.slt?plain=1#L110-L115))?
```
# query D
# SELECT make_date('2024.7', '1', '1');
# ----
# 2024-01-01
```
The `int32_coercion()` comment ([`date_part_name.rs`
L38-L40](https://github.com/apache/datafusion-spark/blob/98e1f480e5444c9ddf550bf8644e073fe2fb74d2/crates/spark/src/function/datetime/date_part_name.rs#L38-L40))
misses two cases for `date_from_unix_date`. On Spark 4.2.0 with ANSI mode off,
`date_from_unix_date('19000.9')` is `2022-01-08` and
`date_from_unix_date(CAST('NaN' AS DOUBLE))` is `1970-01-01`. Both raise a cast
error here. That list has now been incomplete twice, and the `#[user_doc]`
descriptions carry the details. Could the comment say "which errors in cases
where Spark with ANSI mode off returns a value or NULL" instead of naming each
case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]